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Abstract 


This  document  reports  on  the  Department  of  National  Defence  (DND)  Tactical  Aviation 
Mission  System  Simulation  (TAMSS)  Situation  Awareness  (SA)  project.  The  TAMSS  SA 
project  was  conducted  at  the  Centre  for  Applied  Cognitive  Research  (CACR)  at  the  Carleton 
University.  In  accord  with  the  original  goals  of  this  project,  the  deliverables  included  the 
development  of  a  CH146  Griffon  simulation  capability  at  the  CACR,  the  development  of  a 
theoretical  framework  to  guide  the  evaluation  process,  three  experiments  that  both  assessed  an 
engineering  system  and  a  theoretical  framework,  and  this  document,  which  summarizes  the 
TAMSS  SA  project  and  provides  a  link  to  acquisition  programs  and  to  potential  simulation- 
based  training  applications. 

The  TAMSS  SA  project  provides  a  guide  for  the  implementation  of  simulation-based 
evaluation  on  a  cost-effective  platform.  The  combination  of  the  CSE  framework  and  the 
research-enabling  simulation  environment  that  was  developed  in  the  TAMSS  SA  project  can 
be  used  to  reduce  risk  and  enhance  value  in  acquisition  programs.  Collaboration  among  the 
Carleton  University  CACR  researchers,  including  graduate  students,  and  from  the  visits  from 
many  DND  personnel,  subject  matter  experts,  and  industry  representatives,  has  demonstrated 
the  value  of  locating  these  activities  in  a  research-rich  environment. 
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Executive  summary 


Introduction 

This  document  reports  on  the  Department  of  National  Defence  (DND)  Tactical  Aviation 
Mission  System  Simulation  (TAMSS)  Situation  Awareness  (SA)  project.  The  TAMSS  SA 
project  was  conducted  at  the  Centre  for  Applied  Cognitive  Research  (CACR)  at  the  Carleton 
University.  In  accord  with  the  original  goals  of  this  project,  the  deliverables  included  the 
development  of  a  CH146  Griffon  simulation  capability  at  the  CACR,  the  development  of  a 
theoretical  framework  to  guide  the  evaluation  process,  three  experiments  that  both  assessed  an 
engineering  system  and  a  theoretical  framework,  and  this  document,  which  summarizes  the 
TAMSS  SA  project  and  provides  a  link  to  acquisition  programs  and  to  potential  simulation- 
based  training  applications. 

Carleton  University  CH-146  Simulation  Environment. 

The  Carleton  University  CACR  CH-146  simulator  was  developed  as  a  modified  version  of  a 
TAMSS  Networked  Tactical  Simulator  (NTS).  The  CACR  simulator  provides  state-of-the  art 
capabilities  for  experimentation  and  data  collection.  The  simulator  includes  an  out-the- 
window  display,  a  helmet  mounted  display,  head-tracking  capabilities,  a  scenario  generation 
utility,  and  custom  data  collection  capabilities.  In  addition,  an  Electro-optical 
Reconnaissance,  Surveillance,  Targeting  and  Acquisition  (ERSTA)-like  simulation  model  was 
integrated  into  the  CACR  simulator  environment  using  high-level  architecture  (HLA).  The 
CACR  simulator  is  an  effective  platform  for  prototyping  and  exercising  human-machine 
systems  and  for  measuring  the  impact  of  new  technologies  in  a  dynamic  simulation 
environment. 

Theoretical  Framework:  Cognitive  Systems 
Engineering. 

A  survey  of  existing  literature  on  experimental  approaches  to  evaluating  modelling  and 
simulation  allowed  the  development  of  a  Cognitive  Systems  Engineering  (CSE)  framework 
that  combined  three  central  constructs  in  the  field  of  human-machine  collaboration:  situation 
awareness,  workload,  and  task-relevant  performance.  Workload  refers  to  the  cognitive  effort 
required  by  the  operator,  situation  awareness  refers  to  the  operator’s  ability  to  represent  and 
monitor  the  ongoing  activity,  and  task-relevant  performance  refers  to  the  specific  aspects  of 
the  operators’  behaviour  that  relate  to  the  machine  being  evaluated.  All  three  aspects  of  the 
human-machine  system  can  be  evaluated  objectively  and  subjectively  (the  latter  from  the 
perspective  of  the  operator).  Objective  measures  of  performance  can  include  head  position, 
aircraft  characteristics  such  as  heading,  speed,  and  altitude;  of  situation  awareness  -  detection 
of  relevant  objects  in  the  environment;  of  workload  -  response  to  visual  or  auditory  cues. 
Subjective  measures  include  ratings  (Likert  type  and  NASA  TLX  measures  were  both  used) 
of  situation  awareness,  performance,  and  workload.  Various  combinations  of  these  measures 
(with  at  least  one  of  each  category)  were  used  in  each  of  the  three  experiments. 


Experiments. 

In  three  experiments,  the  usefulness  of  the  CSE  framework  for  evaluating  human-machine 
systems  was  demonstrated.  In  Experiment  1,  pilots  showed  reduced  situation  awareness  and 
increased  workload  when  they  were  using  night  vision  goggles.  In  Experiment  2,  ERSTA- 
like  sensor  capabilities,  combined  with  a  digital  moving  map,  allowed  aircrew  to  have 
enhanced  situation  awareness  and  at  times  reduced  workload.  In  Experiment  3,  the  presence 
of  an  mission  specialist  to  operate  an  ERSTA-like  system  allowed  the  mission  commander  to 
have  increased  OTW  viewing  time,  substantially  decreased  workload  (both  objective  and 
subjective),  and  enhanced  situation  awareness.  All  three  experiments  used  experienced  flight 
crews,  and  realistic  missions.  The  findings  supported  the  use  of  the  CSE  framework  for 
modelling  and  simulation  and  for  simulation-based  acquisition  programs. 

CSE-Based  Modelling  and  Simulation  for  Acquisition. 

The  CSE  framework  that  was  developed  in  the  TAMSS  SA  project  provides  a  structure  for 
three  modelling  and  simulation  activities:  design,  rapid  prototyping  and  simulation-based 
evaluation.  For  design  and  rapid  prototyping,  the  constructs  in  the  CSE  framework  orient 
activity  toward  consideration  of  how  the  human- machine  system  will  potentially  affect 
operator  SA,  workload  and  task-relevant  performance.  For  simulation-based  evaluation,  the 
CSE  framework  provides  both  a  conceptual  structure  and  strong  methodological  guidance. 

Integration  into  Simulation-Based  Training. 

Throughout  the  conduct  of  the  work  associated  with  the  establishment  of  a  CH-146  simulation 
environment  at  the  Carleton  University  CACR,  as  well  as  throughout  the  conduct  of 
experiments,  it  became  readily  apparent  that  NTS-like  devices  had  the  potential  to  serve  a  far 
broader  range  of  applications  than  those  exercised  during  the  performance  of  the  TAMSS  SA 
project.  Comments  from  industry,  government  and  subject  matter  experts  suggested  that 
upgraded  NTS-like  devices  be  used  to  augment  Part-Task  Training  (PTT),  Cockpit  Procedures 
Training  (CPT),  Tactics  Training  and  Mission  Rehearsal  requirements.  It  is  concluded  that 
NTS-like  systems  could  contributed  to  a  vertically  integrated  training  solution,  such  as  that 
represented  by  the  Integrated  Simulation  Training  System  (ISTS)  concept  currently 
circulating  within  the  Canadian  Forces. 


Conclusions 

The  TAMSS  SA  project  provides  a  guide  for  the  implementation  of  simulation-based 
evaluation  on  a  cost-effective  platform.  The  combination  of  the  CSE  framework  and  the 
research-enabling  simulation  environment  that  was  developed  in  the  TAMSS  SA  project  can 
be  used  to  reduce  risk  and  enhance  value  in  acquisition  programs.  Collaboration  among  the 
Carleton  University  CACR  researchers,  including  graduate  students,  and  from  the  visits  from 
many  DND  personnel,  subject  matter  experts,  and  industry  representatives,  has  demonstrated 
the  value  of  locating  these  activities  in  a  research-rich  environment. 


Herdman,  C.  M.,  LeFevre,  J.  2004.  Tactical  Aviation  Mission  System  Simulation 
Situational  Awareness  Project.  Defence  R&D  Canada. 
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1.  Project  Overview 


1.1  Major  Tasks 

This  document  reports  on  the  Department  of  National  Defence  (DND)  Tactical  Aviation 
Mission  System  Simulation  (TAMSS)  Situation  Awareness  (SA)  project.  The  TAMSS  SA 
project  was  conducted  at  the  Carleton  University  Centre  for  Applied  Cognitive  Research 
(CACR). 

Two  major  activities  were  completed  in  the  TAMSS  SA  project.  The  first  major  activity  was 
to  establish  a  research-enabling  CH-146  simulation  environment  at  the  Carleton  University 
CACR.  The  second  major  activity  was  to  conduct  three  graduate-level  research  studies  on 
Situation  Awareness  (SA). 

1.1.1  Carleton  University  CACR  CH-146  Simulation  Environment 

The  first  major  task  in  the  TAMSS  SA  project  was  to  establish  a  research¬ 
enabling  CH-146  simulation  environment  at  the  Carleton  University  CACR 
(see  Figure  1).  This  task  was  accomplished  using  a  modified  version  of  a 
TAMSS  Networked  Tactical  Simulator  (NTS)  device  that  included  an  Out- 
The- Window  (OTW)  display,  a  Helmet  Mounted  Display  (HMD),  head¬ 
tracking  capabilities,  a  scenario  generation  utility,  and  custom  data  collection 
capabilities.  In  addition,  an  Electro-optical  Reconnaissance,  Surveillance, 
Targeting  and  Acquisition  (ERSTA)-like  simulation  model  was  integrated 
into  the  CACR  simulator  environment  using  high-level  architecture  (HLA). 

The  Carleton  University  CACR  CH-146  simulation  environment  was 
developed  across  the  extent  of  the  TAMSS  SA  project.  This  cost  effective 
and  malleable  environment  was  proven  to  be  an  effective  platform  for 
prototyping  and  exercising  systems  and  for  measuring  the  impact  of  new 
technologies  in  a  dynamic  simulation  environment.  A  full  description  of  the 
Carleton  University  CACR  CH-146  simulation  environment  is  given  in 
Section  2.1.1. 
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Figure  1  -  CACR  CH-146  Simulation  Environment 

1.1.2  Three  graduate-level  studies 

The  second  major  task  in  the  TAMSS  SA  project  was  to  conduct  three 
graduate-level  studies  on  SA.  The  intent  of  the  studies  was  to  develop  and 
document  methods  for  measuring  SA  in  a  simulation  environment. 

An  analysis  of  the  research  literature  showed  that  human-machine 
interactions  are  often  too  complex  for  a  single  concept  (such  as  SA)  to 
provide  sufficient  information  regarding  the  impact  of  an  interface  on  an 
operator’s  overall  behaviour.  Accordingly,  a  broader  Cognitive  Systems 
Engineering  (CSE)  framework  was  developed  that  combines  three  central 
constructs  in  the  field  of  human- machine  collaboration:  situation  awareness, 
workload,  and  task-relevant  performance.  Situation  awareness  refers  to  the 
operator’s  ability  to  cognitively  represent,  monitor  and  predict  activities, 
workload  refers  to  the  cognitive  effort  required  by  the  operator,  and  task¬ 
relevant  performance  refers  to  the  specific  aspects  of  the  operators’  behaviour 
that  relate  to  the  system  being  evaluated.  The  CSE  framework  was  used  to 
guide  investigations  in  the  three  TAMSS  experiments. 

The  effectiveness  of  the  CSE  framework  was  examined  and  proven  in  the 
three  TAMSS  experiments.  In  all  three  studies,  trained  CEI-146  aircrews 
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were  required  to  complete  simulated  missions  (e.g.,  reconnaissance)  in  the 
Carleton  University  flight  simulator.  In  Experiment  1,  CH-146  pilots  showed 
reduced  situation  awareness  and  increased  workload  when  they  were  using  a 
heads-up  display.  In  Experiment  2,  ERSTA-like  sensor  capabilities, 
combined  with  a  digital  moving  map,  allowed  CH-146  aircrew  to  have 
enhanced  situation  awareness  and  at  times  reduced  workload.  In  Experiment 
3,  it  was  shown  that  the  presence  of  an  mission  specialist  to  operate  an 
ERSTA-like  sensor  system  allowed  the  CH-146  mission  commander  to  have 
increased  OTW  viewing  time,  substantially  decreased  workload  (both 
objective  and  subjective),  and  enhanced  situation  awareness. 

The  three  studies  in  the  TAMSS  SA  project  unequivocally  show  that  the  CSE 
framework  provides  an  effective  guide  for  assessing  human-machine  systems 
in  a  simulation-based  environment.  The  effectiveness  of  the  CSE  framework 
is  enhanced  when  an  evaluation  includes  at  least  one,  and  preferably  multiple 
measures  of  situation  awareness,  workload,  and  of  task-relevant  performance. 
The  use  of  multiple  measures  will  allow  for  a  richer  and  more  accurate  index 
of  how  new  technologies  affect  the  human-machine  interaction.  As 
discussed  in  Section  2.2,  the  CSE  framework  can  provide  an  effective 
structure  for  simulation-based  evaluation  activities  in  DND  acquisition 
programs. 
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2.  TAMSS  SA  Project  Activities 


2.1  Carleton  University  Simulation  Environment 

A  major  activity  in  the  TAMSS  SA  project  was  to  develop  a  research-enabling  CH-146 
simulation  environment  at  the  Carleton  University  CACR  lab.  This  activity  spanned  across 
the  full  TAMSS  SA  project. 

2.1.1  Overview 

The  flight  simulator  at  Carleton  University  is  a  custom  version  of  the 
Networked  Tactical  Simulator  (NTS)  that  has  been  developed  by  The  HFE 
Group,  Inc.  as  part  of  the  TAMSS  initiative.  As  with  other  NTS  systems,  the 
Carleton  NTS  represents  the  flight  deck,  mission  equipment,  and  physical 
structure  of  the  CH-146  Griffon  helicopter  flown  by  the  Department  of 
National  Defence  (DND).  The  Carleton  simulator  includes  both  out-the- 
window  (OTW)  and  Helmet  Mounted  Display  (HMD)  capabilities.  The 
Carleton  simulator  has  a  head-tracker  to  support  the  use  of  the  HMD. 
Additionally,  the  simulator  supports  the  creation  of  synthetic  enviromnents 
and  scenario  creation  via  staging  software. 

The  Carleton  NTS  is  unique  in  that  it  includes  experimental  and  data 
collection  capabilities.  These  experimental  capabilities  allow  a  user  to  create 
visual  and  auditory  events  that  can  be  inserted  into  a  mission.  The  data 
collection  capabilities  enable  an  experimenter  to  examine  over  100  logged 
measures  in  analyzing  the  performance,  workload,  and  situation  awareness  of 
the  pilots  flying  the  simulator. 

The  Carleton  NTS  consists  of  six  PCs,  running  the  Windows  2000 
Professional  operating  system.  Three  of  the  PCs  are  used  for  image 
generation  (IG1,  IG2,  and  IG3)  and  simultaneously  project  onto  three  8’  x  6’ 
screens,  providing  the  pilot  with  a  near  180-degree  horizontal  and  40-degree 
vertical  view.  Two  PCs  (INSTR1,  INSTR2)  are  used  for  simulation  of  the 
flight  model  and  instrumentation.  INSTR1  is  responsible  for  running  the 
helicopter  flight  model  (HELISIM),  simulating  the  avionics,  and  for  driving 
the  pilot  instrument  panel.  INSTR2  is  responsible  for  the  operation  of  the 
CDUs.  As  well,  the  INSTR1  PC  hosts  the  custom  data  collection  software 
used  in  the  Carleton  experiments.  Finally,  the  sixth  PC,  the  Experimenter 
Operating  Station  (EOS),  is  responsible  for  overall  system  control,  including 
mission  loading  and  unloading.  As  well,  this  PC  hosts  the  scenario 
generation  software  and  a  Stealth  viewer.  The  simulator  includes  an  ASTi 
Digital  Audio  Communications  System  (DACS)  that  supports  simulation  of 
cockpit  voice  communication  as  well  as  voice  communication  between  the 
pilot  and  console  operator.  Data  transfer  between  the  various  modules  occurs 
via  one  of  three  modes.  High  volume  data,  such  as  that  between  the  flight 
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model  software  and  scene  generator,  uses  UDP  communications. 
Communications  between  the  avionics  simulation,  the  pilot  instrument  panels 
and  the  CDU  infrastructure  is  via  high-level  architecture  (HLA).  HLA  is  also 
used  to  interface  the  simulator  to  external  systems.  Finally,  shared  memory  is 
used  for  communications  between  the  CDU  and  the  CDU  Proxy,  which 
facilitates  integration  of  the  non-FILA  native  CDU  simulation  into  the  NTS 
federation.  Figure  2  and  Figure  3  provide  a  general  overview  of  the 
hardware,  functionality,  and  communications  infrastructure  of  the  Carleton 
NTS. 

2.1.2  Simulation  Hardware 

The  following  summarizes  the  main  components  of  the  Carleton  NTS.  The 
six  PCS  in  the  Carleton  simulator  are  equipped  with  dual  Pentium  III  1GHz 
processors.  Each  machine  has  1  GB  of  RAM.  The  machines  are  physically 
networked  using  a  3Com  SuperStack  3  Baseline  10/100  12  port  Ethernet 
Switch.  Three  NEC  Model  MT1055  Data  Projectors  are  used  to  project 
generated  images  (using  Vega  software)  on  8’x  6’  screens,  creating  the 
immersive  out-the-window  scene.  The  head-tracking  system  used  is  an 
Intersense  IS-900  Virtual  Workbench  Tracking  System.  The  IS-900  is  a  6 
degrees-of- freedom  tracker,  tracking  both  position  and  angular  changes  (X, 

Y,  Z,  Heading,  Pitch  and  Roll).  The  IS-900  provides  position  resolution  of 
1 .5mm  in  position  and  an  angular  resolution  of  0.05  degrees.  It  is  jitter-free 
with  a  position  stability  of  4  mm  and  angular  stability  of  0.2,  0.4  RMS.  For  a 
detailed  depiction  of  the  hardware  configuration  of  the  Carleton  simulator, 
refer  to  Figure  3. 
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Diagram  provided  courtesy  of  the  HFE  Group 

Figure  2  -  Carleton  Simulator  General  Overview 
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Diagram  provided  courtesy  of  the  HFE  Group 

Figure  3  -  Carleton  Simulator  Hardware  Overview 

2.1.3  Software  Overview 

2. 1.3. 1  COTS  Products 

The  simulator  software  consists  of  several  Commercial  Off-the- 
Shelf  (COTS)  Software  packages  integrated  by  custom  control, 
communications,  and  experimental  software.  The  primary  COTS 
products  used  and  their  function  within  the  Carleton  system  are  as 
follows: 

Vega.  Vega,  a  product  from  Multigen-Paradigm  is  the  COTS  tool 
used  to  render  the  Out  the  Window  (OTW)  or  through  the  HMD 
external  scene.  Vega’s  strength  is  that  it  is  able  to  render  complex 
geometries  in  real-time.  It  is  a  key  component  in  achieving  visual 
realism  in  the  simulation.  It  is  ultimately  based  on  the  OpenGL 
2D  and  3D  graphics  application  programming  interface  (API). 

Vega  uses  Openflight  (a  3D  file  format)  models  of  both  the  terrain 
database  and  scene  objects  to  render  the  scene.  These  entities  are 
configured  into  the  Vega  application  using  the  LynX  graphical 
interface.  This  graphical  interface  is  used  to  create  Vega 


7 


application  definition  files  (.adf  files).  These  files  describe  both 
graphical  and  platform  related  details  of  the  Vega  application. 
Vega  renders  the  outside  scene  based  on  the  graphical  objects 
defined  in  the  .adf  files  and  a  given  “eye”  point  determined  by  the 
aircraft  position  and/or  head  position. 

Vega  also  includes  a  development  API  that  enables  a  user  to 
customize  Vega  functionality  for  specific  applications.  For 
example,  this  API  is  used  in  the  Carleton  system  to  generate  HUD 
information  and  symbology.  Vega  software  callbacks  are  used  to 
superimpose  the  HUD  information  on  the  Vega  scene  as  seen 
through  Night  Vision  Goggles  (NVGs).  The  API  has  also  been 
used  to  extend  Vega  capabilities  to  handle  visual  experiment 
events  generated  through  the  experimental  software. 

HELISIM.  Helisim,  from  Virtual  Prototypes  Incorporated,  is  a 
software  package  used  to  provide  the  flight  model.  HELISIM 
mimics  the  performance  of  a  rotary-wing  aircraft  by  tuning 
parameters  such  as  weight  and  balance,  propulsion  and  rotor 
characteristics,  and  instrumentation,  thus  enabling  the  simulator  to 
closely  represent  the  flight  dynamics  of  the  CH-146  Griffon. 
HELISIM  accepts  inputs  from  the  collective,  cyclic  and  pedals  of 
the  simulator  and  using  the  defined  flight  model,  updates  aircraft 
position  (i.e.,  latitude,  longitude,  and  altitude),  aircraft  heading, 
pitch,  and  roll  as  well  as  several  other  flight  and  instrumentation 
values. 

An  important  feature  of  HELISIM  is  an  API  that  allows  for  real¬ 
time  control  of  various  aircraft  parameters.  This  is  an  extremely 
important  feature  for  experimentation.  Using  these  HELISIM 
features,  the  Carleton  lab  has  developed  a  capability  to  freeze 
specific  instrumentation  (e.g.,  aircraft  heading,  radar  altimeter, 
etc.).  Pilots'  situation  awareness  of  their  cockpit  systems  is 
measured  using  the  freezing  capability  (e.g.,  did  the  pilot  notice 
the  frozen  instrumentation,  how  long  did  it  take  for  them  to  notice 
and  react  to  the  frozen  instrumentation).  The  Carleton  lab  has  also 
used  the  HELISIM  API  to  develop  a  capability  to  measure  the 
control  of  aircraft  position  and  orientation  based  on  an  ADS-33 
attitude  recovery  task. 

STAGE.  STAGE  is  the  acronym  for  Scenario,  Toolkit  and 
Generation  Environment.  It  is  a  software  tool  used  to  create 
complex  tactical  scenarios.  STAGE  provides  a  graphical  user 
interface  in  which  to  enter  information  into  a  tactical  database. 
This  database  then  generates  the  real-time  tactical  scenario. 
STAGE  also  displays  the  real-time  positions  of  entities  in  the 
scenario  as  it  is  run  on  its  situation  display. 
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STAGE  is  used  to  add  “entities”  to  the  simulated  mission 
scenarios.  This  STAGE  entity  information  is  sent  to  Vega,  which 
renders  the  STAGE  entities  in  the  external  scene  in  the 
appropriate  position.  The  level  at  which  the  pilot  detects  the 
STAGE  entities  during  the  mission  can  be  used  to  gauge  the 
pilot’s  level  of  situation  awareness. 

STAGE  can  be  run  in  one  of  two  modes  -  with  EILA  enabled  or 
disabled.  When  EILA  is  enabled,  STAGE  becomes  the  HLA 
gateway  for  the  entire  system  and  can  be  used  to  send  the  STAGE 
entity  (including  Ownship)  information  to  external  agencies. 

When  STAGE’S  EILA  is  not  enabled,  STAGE  communicates  only 
with  the  other  simulator  components. 

STEALTH.  The  MAK  Stealth  viewer  is  a  3D  visualization  tool 
that  extends  the  console  operator's  viewpoint  of  the  simulated 
environment  beyond  the  fixed  point  of  the  pilot  to  anywhere  in  the 
simulated  world.  Stealth  enables  the  console  operator  to  attach  to 
other  entities  in  the  simulated  environment  to  see  the  world 
through  their  eyes.  Stealth  receives  its  information  on  entity 
position  from  STAGE  using  the  HLA  interface. 

2. 1.3.2  Custom  Software  Code 

Custom  code  within  the  Carleton  simulator  is  used  for  the 
Experimenter  Operator  Station  (EOS).  EOS  is  responsible  for  the 
command  and  control  of  various  components  as  well  as  for  the 
unique-to-Carleton  experimental  and  data  logging  capabilities.  A 
more  detailed  description  of  the  EOS  capabilities  is  given  in  the 
next  section.  Custom  code  is  also  used  to  develop  the  HMD 
symbology  generation  capabilities. 

Other  components  that  use  custom  code  are  the  avionics 
simulation  module,  the  pilot  Head-Down  Display  (HDD) 
instrument  panel,  the  CDU  bezel  and  CDUs,  and  the  HUD 
symbology.  The  Communications  software  within  the  simulator, 
whether  via  HLA,  UDP  or  Shared  Memory  is  also  custom  code. 
Tools  used  in  the  development  of  the  custom  code  include  VAPS 
by  Virtual  Prototypes  Incorporated  (CDU),  GLStudio  (pilot  head- 
down  instrument  panel  and  HUD  symbology)  by  DiSTI,  and  VR- 
Link  (HLA)  by  MAK  Technologies. 

2.1.4  The  Experimenter  Operator  Station  (EOS) 

The  EOS  encapsulates  control  functions  for  the  Carleton  simulator  as  well  as 

the  experimental  capabilities  are  unique  to  the  Carleton  CACR  simulation 

system. 
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2. 1.4. 1  Command  and  Control  Capabilities  of  the  EOS 

The  control  functions  of  the  simulator  are  accessed  via  a  GUI 
interface.  The  main  control  functions  are  as  follows: 

1.  The  Mission  Control  function  opens,  loads,  resets,  unloads 
and  exits  missions. 

2.  The  Location  function  allows  for  console  control  of  the 
aircraft  position.  It  has  a  slew  mode  in  which  the  operator  can 
move  the  aircraft  in  all  cardinal  directions  as  well  as  change 
the  aircrafts  altitude  and  heading. 

3.  The  Weather  function  changes  atmospheric  conditions  such  as 
wind  direction,  wind  speed  and  cloud  cover  in  the  OTW 
scene. 

4.  The  Ownship  function  can  be  used  to  alter  the  ownship  fuel 
levels,  communications  and  navigation  settings. 

5.  The  Options  function  currently  enables  the  operator  to  change 
the  date  and  time  at  which  the  mission  is  taking  place.  The 
Vega  generated  scene  will  accordingly  adjust  lighting  when 
these  are  changed. 

6.  The  Communications  function  enables  the  operator  to  monitor 
and  transmit  on  a  specified  frequency  via  the  Digital  Audio 
Communication  System  (DACS).  It  also  has  an  intercom 
capability. 

7.  The  Freeze  function  allows  the  operator  to  freeze  the  aircraft, 
the  scenario  or  both  (freeze  all). 

8.  The  System  Control  function  gives  the  operator  some  degree 
of  remote  control  over  the  PCs  in  the  network.  It  supports 
system  Reboot,  Reboot  IGs,  and  system  Shutdown 
capabilities. 

2. 1.4.2  Experimental  Capabilities  of  the  EOS 

The  experimental  capabilities  of  the  Carleton  simulator  make  it 
different  from  the  average  flight  simulator.  The  Carleton  system 
has  the  ability  to  generate  events,  log  pilots'  responses  to  events, 
and  log  key  flight-related  data  throughout  a  mission.  This  data 
can  then  be  used  to  assess  pilot  performance,  workload  and 
situation  awareness. 
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The  experimental  software  has  three  major  components: 
Experiment  Scenario  Generation,  Experiment  Control,  and  Data 
Collection.  The  interplay  of  these  three  components  is  shown  in 
Figure  4. 

2. 1.4. 3  Experiment  Scenario  Generation 

The  Experiment  Scenario  Generation  creates  an  experimental 
scenario  by  defining  a  list  of  events  that  are  to  occur  during  the 
course  of  the  mission.  The  system  currently  supports  the 
generation  of  audio  and  visual  events.  Audio  events  generate  a 
tone  of  specified  frequency  and  duration  using  the  DACS.  Visual 
events  generate  an  image  that  is  displayed  in  Vega  for  a  specified 
duration  of  time.  The  experimenter  can  set  the  onset  time,  timing 
mode  (periodic  or  on-shot  events),  as  well  as  the  duration  of  the 
probe.  For  periodic  timing,  the  experimenter  specifies  the  period 
interval  as  well  as  the  variance  in  interval  time.  For  visual  events, 
the  experimenter  can  also  specify  the  reference  point  (e.g.,  Flead, 
Aircraft,  or  World  reference),  as  well  as  the  relative  position  to  the 
frame  center  (e.g.,  X,  Y,  Z,  Fleading,  Pitch,  Roll)  at  which  the 
event  should  appear. 
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Trigger 


Diagram  provided  courtesy  of  the  HFE  Group 


Figure  4  -  Experiment  System  Overview 


There  are  two  types  of  tasks  associated  with  the  event,  detection 
and  discrimination,  which  are  currently  supported  by  the  system. 
In  the  detection  task,  the  pilot’s  task  is  to  detect  audio  or  visual 
stimuli  and  respond  as  quickly  as  possible  to  the  stimuli,  usually 
by  pressing  a  button.  In  the  discrimination  task,  the  pilot's  goal  is 
to  identify  a  specific  object  out  of  a  range  of  objects,  using  a 
TRUE/FALSE  type  of  response.  The  user  is  able  to  specify  the 
number  of  objects  to  be  used  in  the  experiment  as  well  as  the 
weighting  of  TRUE  to  FALSE  responses. 

Once  all  the  events  have  been  specified,  the  experimental  scenario 
is  saved  as  an  “.exp”  file,  and  the  experiment  is  ready  to  be 
launched. 
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2.1. 4.4  Experiment  Control 


Experiment  Control  launches  the  experimental  scenario  and  is 
used  to  initiate  the  data  collection  process  once  the  desired  “.exp” 
file  has  been  selected  by  the  experimenter.  A  graphical  interface 
enables  the  user  to  select,  launch  and  stop  an  experimental 
scenario.  The  EOS  feeds  information  from  the  “.exp”  file  to  an 
event  scheduler  which  initiates  the  audio  and  visual  events  at  the 
designated  times.  Experiment  Control  also  contains  a  head  tracker 
calibration  utility. 

2. 1.4.5  Data  Collection  Capabilities 

Data  Collection  is  a  background  process  that  starts  automatically 
when  the  experiment  is  launched.  The  Data  collection  application 
logs  the  following  information: 

1.  Session  information:  Session  ID,  Start  Time,  End  Time, 
Subject  Number,  Trial  Number 

2.  Audio  Events:  Session  ID,  Simulator  Time  (of  event),  Task 
Type,  Event  Duration,  Truth  Value,  IsTarget  (in 
discrimination  task),  Tone  Frequency 

3.  Visual  Events:  Session  ID,  Simulator  Time  (of  event),  Task 
Type,  Event  Duration,  Truth  Value,  Is  Target  (in 
discrimination  task),  Object  Number,  Reference  (Head, 
Aircraft,  World),  X,  Y,  Z,  H,  P,  R 

4.  Button  Press:  Session  ID,  Simulator  Time  (of  button  press), 
Button  Number  Pressed 

5.  Head-tracker  Data:  X,  Y,  Z,  H,  P,  R  (Positional  and  Angular 
information) 

6.  HELISIM  Data:  Session  ID,  Simulator  Time  (of  data)  plus  99 
other  parameters  derived  by  HELISIM.  The  most  important 
of  these  for  experimental  purposes  are:  Latitude,  Longitude, 
Altitude,  Speed  and  Heading. 

The  above  items  are  logged  in  separate  SQL  tables  and  can  be 
examined  and  extracted  using  a  Microsoft  Access  application  as  a 
front-end  to  the  database. 
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2.1.5  Experimental  Features  Added 


In  addition  to  the  experimental  features  provided  by  The  HFE  group,  the 
CACR  has  successfully  added  several  new  experimental  features  to  the 
simulator.  These  include  add-ons  and  upgrades  to  The  HFE  Group’s  original 
applications,  as  well  as  new  applications  allowing  for  new  experimental 
designs  and  measurement.  This  section  will  briefly  review  these  features. 

2. 1.5. 1  Freeze  HDD  instruments 

An  application  has  been  developed  to  give  the  operator  of  the 
experiment  the  ability  to  freeze  at  any  given  moment  any  of  the 
following  indications  displayed  on  the  HDD:  Air  speed.  Heading, 
Radar  Altitude,  Barometric  Altitude  and  Torque.  This  feature  also 
records  the  time  at  which  the  freeze  command  was  initiated, 
allowing  the  experimenters  to  refer  to  this  information  at  a  later 
time.  This  feature  also  provides  the  operator  with  the  ability  to 
remove  and  replace  the  entire  HUD  display  upon  command. 

2. 1.5.2  Attitude  recovery  task 

Although  not  used  in  the  TAMSS  SA  experiments,  a  complete 
attitude  recovery  task  was  created.  The  task  consists  of  blanking 
the  screen,  placing  the  aircraft  at  a  pre-configured  attitude,  then 
un-blanking  the  screen  and  passing  control  to  the  pilot.  The  pilot 
is  then  instructed  to  indicate  their  completion  of  recovery  by 
pressing  one  of  the  cyclic  buttons.  During  the  recovery  process, 
numerous  data  are  recorded  (in  addition  to  aircraft  attitude) 
including  the  time  at  which  the  scene  was  un-blanked  and  control 
was  passed  to  pilot  and  the  time  at  which  the  pilot  indicated  task 
completion. 

This  application  also  allows  for  the  task  difficulty  to  be  controlled 
by  the  experimental  operator.  This  control  can  be  achieved  by 
controlling/configuring  the  initial  aircraft  attitude  and  by 
controlling/configuring  the  turbulence  at  the  area  where  the  task 
takes  place.  In  addition,  the  following  task  parameters  can  be 
configured  by  the  experimental  operator  through  the  use  of  a  task 
configuration  table  (a  simple  text  file):  Air  speed.  Altitude, 
Heading,  Pitch,  Roll,  Torque  (this  is  optional  in  cases  when 
inexperienced  pilots  are  being  used),  Turbulence  (vertical  air 
speed,  horizontal  air  speed,  period  of  cycles). 

2. 1.5.3  Cyclic  button  press  recording  and  filtering 

A  simple  filtering  mechanism  has  been  added  to  clear  the  noise 
that  is  recorded  when  a  pilot  presses  and  holds  down  the  cyclic 
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button  for  a  long  period  of  time.  A  thread  was  added  to  allow  for 
a  cyclic  button  press  sampling  rate  of  5  ms.  This  sampling  rate 
can  be  adjusted  as  necessary. 

2.1.6  General  Additions  and  Upgrades 

An  interface  for  HELISIM  was  added  to  the  CACR  simulator  to  allow 
developers  to  fly  the  aircraft  from  the  development  station  using  a  simple 
joystick  while  using/monitoring  other  tools  at  the  development  station.  The 
addition  of  these  experimental  features  allows  the  CACR  team  to  measure  the 
SA  of  pilots  under  several  new  conditions  (i.e.,  when  the  HUD  or  HDD 
freezes,  or  when  they  must  recover  after  losing  control  of  the  aircraft).  These 
measures  will  assist  in  the  definition  and  measurement  of  SA. 

2.1.7  Data  Collection:  Evaluation  and  Testing 

The  Carleton  simulator  is  used  for  experimentation,  and  thus  it  is  important  to 
determine  the  amount  of  time  delay  occurring  between  a  pilot's  action  or 
response  in  the  simulator  and  the  logging  of  this  response  by  the  experimental 
software.  Ideally  this  delay  will  be  minimal  so  that  the  simulator  mimics 
real-time  events  as  closely  as  possible.  Furthermore,  the  amount  of  time  that 
it  takes  for  information  to  pass  through  the  system  to  the  EOS  should  be 
consistent  over  time  to  ensure  that  the  measures  being  sampled  are  accurate. 
The  next  section  describes  the  tests  that  have  been  performed  by  the  CACR  in 
an  attempt  to  measure  the  time  delays  occurring  within  the  Carleton 
simulator. 

2. 1.7.1  UDP  packet  delay  measures 

The  CACR  engineers  have  taken  measurements  to  determine  the 
amount  of  time  required  to  transfer  a  single  UDP/IP  packet 
between  two  applications  running  on  two  separate  PCs.  UDP/IP 
packets  are  the  main  data  transfer  mechanism  used  by  the 
simulator.  They  are  used  both  for  direct  communication  between 
the  several  modules  that  make  up  the  simulator  system  and  as  a 
transport  layer  for  “over  HLA”  communication.  The  delay  of  a 
single  small  UDP  packet  sent  between  two  applications  running 
on  two  separate  machines  was  measured  by  the  CACR.  This 
measurement  was  gained  using  loop-back  methodology  and  was 
based  on  the  internal  Windows  clock  giving  a  resolution  of  1ms. 

It  was  found  that  the  measured  delay  in  a  loaded  system  at  a 
steady  state  (while  executing  standard  Out  the  Window  scene)  is 
always  less  than  2  ms. 
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2. 1. 7.2  Serial  port  loop-back  delay 


The  delay  between  the  writing  and  the  reading  of  a  single  ASCII 
character  through  the  PCs  serial  port  was  measured  by  means  of 
an  external  loop-back.  When  a  single  ASCII  character  is  written 
to  one  of  the  PC's  serial  ports,  it  triggers  the  digital  scope.  The 
digital  scope  is  used  for  several  timing/delay  measurements  in  a 
system. 

In  order  to  perform  this  measurement,  the  Tx  pin  of  the  serial  port 
was  shortened  externally  with  the  Rx  pin.  Then,  a  Windows 
application  was  used  to  send  a  single  character  (8  bit)  at  115600 
bits/sec  using  the  standard  Windows  WriteFile()  call  and  receive 
the  character  using  the  standard  Windows  blocking  ReadFile() 
call.  The  delay  between  the  two  calls  was  measured  using  the 
internal  Windows  clock  with  a  resolution  of  1ms.  The  measured 
delay  between  the  two  calls  was  found  to  be  less  than  2  ms.  This 
measurement  gives  us  an  indication  of  the  interaction  delays  in 
our  system  between  the  application  layer  and  the  hardware  layer. 
These  interactions  are  carried  out  by  the  Windows  device  drivers. 

2. 1.7.3  Vega  timing  (for  post-draw  rendering) 

This  measurement  was  taken  to  determine  the  amount  of  delay 
that  occurs  between  the  time  that  an  object  is  added  to  the  post¬ 
draw  function  in  Vega  and  the  time  at  which  the  object  appears  on 
the  projection  screen.  The  last  stage  at  which  Vega  can  execute  a 
user  code  before  the  display  buffers  for  the  current  frame  are 
swapped  and  then  displayed,  is  during  the  post-draw  callback 
stage.  The  user  can  add  visual  objects  such  as  the  HUD 
symbology  or  any  other  visual  objects  at  this  stage.  The  CACR 
team  measured  this  delay  using  the  following  process: 

1 .  A  method  that  blanked  the  scene  (the  display  buffer)  was 
called. 

2.  A  method  to  draw  a  small  bright  (white)  rectangle  at  the  top 
of  the  scene  was  called. 

3.  A  signal  to  trigger  a  two-channel  digital  scope  was  sent 
through  the  serial  port  to  the  scope. 

4.  Control  was  taken  by  Vega  that  swapped  the  display  buffer. 

This  process  allowed  transmission  the  photo  diode  that  was 
connected  to  the  second  channel  of  the  digital  scope  through  an 
amplifier  to  send  a  signal  to  the  scope  as  soon  as  the  bright 
rectangle  appeared  at  the  top  of  the  screen.  The  delay  between  the 
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time  that  an  object  was  added  as  a  post-draw  object  and  the  time 
that  it  took  to  appear  at  the  top  of  the  screen  (in  this  case  as  a 
white  rectangle)  was  28  ms  (±lms).  When  the  same  bright 
rectangle  was  moved  to  the  bottom  of  the  screen,  the  measured 
delay  was  44  ms  (±lms)  that  is,  28  ms  +  1/60  Hz.  It  is  important 
to  note  that  the  delay  between  the  end  of  the  post-draw  stage  (that 
is  under  the  user’s  code  control)  and  the  time  when  the  actual 
image  starts  to  appear  at  the  top  of  the  screen  should  remain 
constant  regardless  of  the  amount  of  displayed  data. 

2. 1.7. 4  Cyclic  button  press  delay 

The  latency  between  a  button  press  on  the  cyclic  and  the  time  it 
takes  to  receive  this  response  at  the  PCI  based  A/D  board  was 
measured.  In  the  SA  Experiment  1,  a  button  press  on  the  cyclic 
was  used  by  the  pilots  in  response  to  stimuli  that  were  presented 
as  part  of  the  scenario.  Originally  the  pilots'  controls  (including 
the  cyclic  buttons)  were  sampled  by  HELISIM  at  a  rate  of  60  Hz, 
meaning  there  was  a  minimum  delay  of  16.66  ms.  A  separate 
thread  has  been  added  by  the  CAC-R  to  make  the  sampling  rate 
independent  of  HELISIM.  This  has  allowed  for  a  sampling  rate 
much  faster  than  60Hz.  In  future  applications,  the  delay  will  be 
re-measured  with  the  addition  of  this  new  thread.  Sending  a 
character  through  one  of  the  serial  Tx  pins  and  connecting  it 
externally  to  one  of  the  discrete  inputs  on  the  A/D  board  will  gain 
this  measure.  The  time  difference  between  the  call  to  WriteFile() 
and  the  time  that  a  change  in  the  state  of  the  discrete  input  is 
sensed  by  the  thread  will  be  measured  using  the  Windows  clock. 

2. 1.7.5  Time  synchronization  mechanism 

A  time  synchronization  mechanism  to  synchronize  the  6  PCs  in 
the  simulator  has  been  implemented.  This  mechanism  consists  of  a 
timeserver  that  executes  on  one  of  the  PCs  and  time  clients  that 
periodically  (at  ten  second  intervals)  send  time  synchronization 
requests  to  the  server.  These  time  clients  also  receive  updated 
time  responses  that  are  used  to  set  their  PC's  internal  clock. 

As  part  of  this  process,  half  of  the  travel  delay  (the  time  between 
sending  the  sync  request  and  receiving  the  time  sync  replay)  is 
subtracted  at  the  client  side  to  compensate  for  the  communication 
delay  between  the  client  and  the  server  (if  the  measured  round  trip 
delay  is  longer  than  4  ms  the  clock  is  not  getting  updated  during 
that  cycle).  A  UDP/IP  packet  with  a  delay  of  less  then  2  ms  was 
sent  between  the  PCs  to  determine  the  degree  of  synchronicity 
between  the  PC  clocks.  The  clock  synchronization  in  the  system 
was  found  to  be  ±2ms.  This  mechanism  can  be  used  to  obtain 
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accurate  response  measurements  in  the  system,  especially  when  an 
event  is  initiated  by  an  entity  executing  on  one  machine  while  the 
response  for  the  event  is  accepted  by  an  entity  executing  on  a 
different  machine. 

2.1.8  Data  Collection  in  a  Distributed  Environment 

Data  collection  in  a  distributed  simulation  environment  brings  to  bear  a 
number  of  issues.  These  are  discussed  below. 

2. 1.8. 1  Architecture  of  distributed  data  collection  system 

In  the  TAMSS  experiments,  a  distributed  data  collection 
architecture  was  developed  as  shown  in  Figure  5.  All  data  is 
collected  on  a  single  machine  and  logged  into  a  standard  SQL  data 
base.  Each  of  the  data  generating/sampling  tasks  sends  data  over 
a  UDP/IP  socket  to  the  data  collecting  process  at  it’s  own  pace. 
The  data  collecting  process  consists  of  7  threads,  one  per  each 
sending  process.  Each  thread  is  blocked  on  it’s  receive  socket 
until  a  packet  with  data  to  be  logged  arrives.  As  soon  as  the  data 
arrives,  it  is  time-stamped  based  on  the  internal  clock  on  the 
INSTR2  machine  and  then  written  into  the  SQL  data  base.  Beside 
each  data  record  that  is  represented  as  a  row  in  the  relevant  table, 
there  is  a  column  to  store  the  time  stamp  (in  milliseconds)  for  that 
row.  The  time  stamp  represents  elapsed  time  since  the  beginning 
of  the  data  collection  session. 
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Figure  5  -  Data  Collection  in  a  Distributed  (HLA)  Architecture 

2. 1.8.2  Fidelity  of  timing  for  the  logged  data 

Referring  to  Figure  6,  assume  that  a  participant  presses  a  response 
key  at  time  tpress  to  indicate  that  she/he  had  detected  an  event  (e.g., 
audio  or  visual  probe)  that  had  occurred  at  time  tevem.  The 
response  keys  are  sampled  periodically  at  60Flz,  thereby  implying 
a  delay  of  up  to  1/60  seconds  (16.66ms)  between  the  actual  key 
press  and  the  time  the  key  was  sampled,  hence  dsampk  <  16.66ms. 
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At  this  stage,  data  is  written  to  the  sending  socket  and  is  being 
queued  until  the  operating  system’s  kernel  (Windows)  sends  it 
over  UDP  to  the  data  collecting  machine.  This  delay  is  dqsend,  and 
is  hard  to  measure  directly.  The  data  packet  is  transmitted  by  the 
Windows  network  driver  over  Ethernet  through  the  local  switch  to 
the  data  collecting  machine.  The  transmission  delay  for  a  small 
packet  is  d Ethernet  and  is  typically  very  short  when  a  100Mb 
Ethernet  is  used.  The  packet  is  received  at  the  data  collecting  node 
by  the  Windows  network  driver  and  is  being  queued  until  the  data 
collection  process  is  scheduled  by  Windows.  This  delay  is  dscheduUng 
and  is  typically  <  20ms.  Based  on  measurements  using  a  digital 
scope  and  two  Windows  2000  machines  (a  loop  back  test),  dnetwork 
is  the  sum  of  dqsend,  dEthemet  and  dscheduUng  <  22ms.  As  soon  as  the 
packet  is  received  by  the  data  collecting  thread,  the  internal 
hardware  clock  is  sampled  in  order  to  assign  a  time  stamp  to  the 
data.  It  takes  up  to  20ms  to  acquire  the  time  (regardless  if  it’s  the 
time  of  the  day  clock  or  the  better  resolution  multimedia  clock) 
d timestamp'  This  delay  is  due  to  the  fact  that  a  request  to  access  the 
clock  is  queued  by  Windows  and  only  the  Windows  kernel  can 
access  that  clock  and  return  the  result  to  the  requesting  process. 

All  the  above  sums  to  a  maximum  delay  of  d,ampie  +  dnetwork  + 
d timestamp  <  58.66ms  which  is  the  worst  case  resolution  for  the  timing 
data.  For  data  collection  from  HELISIM  or  from  the  head  tracker, 
dsampie  can  be  ignored  and  hence  the  resolution  is  at  42ms. 

2. 1.8.3  Database 

A  relational  data  base  (MySQL)  was  used  to  store  collected  data 
in  the  TAMSS  simulation  system.  This  is  the  most  popular 
database  storage  structure  in  general  use.  Microsoft  Access  was 
used  to  access  MySQL  data  through  an  ODBC  adaptor  using  the 
“C”  programming  language.  During  the  TAMSS  SA  experiments, 
failures  with  the  provided  ODBC  adaptor  resulted  in  some  data 
loss.  These  failures  were  corrected  in  the  later  experiments. 

2. 1.8.4  Collected  data 

Data  is  collected  from  the  following  sources  during  a  simulation 
run: 

1.  HELISIM.  Data  such  as  position,  orientation,  velocity  and 
other  flight  dynamics  parameters  of  the  flight  simulator  were 
sampled  and  logged  at  60Hz. 

2.  Head  tracker  data.  Pitch,  roll,  yaw,  x,  y,  z  values  of  the  head 
mounted  sensor  were  sampled  at  30  Hz. 
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3.  Cyclic  buttons  and  other  response  keys.  All  of  the  cyclic 
buttons  plus  a  foot  switch  were  sampled  at  60Hz 

4.  ERSTA  sensor.  The  ERSTA  sensors  control  modes, 
orientation,  FOV,  zoom,  etc.,  were  sampled  at  60Hz. 

5.  Auditory  and  visual  events  for  generation  of  work  load  were 
recorded  immediately  after  generation. 
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2. 1.8.5  Recommendations  for  future  systems 

The  main  advantage  of  the  current  architecture  for  data  collection 
is  its  simplicity  and  flexibility.  It  is  a  relatively  straight  forward  to 
add  a  data  collection  module  to  collect  new  data  items  from  a  new 
module  such  as  ERSTA.  When  compared  to  other  possible 
architectures  such  as  the  usage  of  HLA  for  data  collection,  the 
current  architecture  probably  generates  the  least  network  traffic 
possible  due  to  the  smallest  possible  overheads  and  hence  small 
and  optimized  data  packets.  The  lean  network  traffic  should 
contribute  to  minimization  of  network  delays  between  the 
participating  nodes  in  the  simulation. 

The  most  evident  disadvantage  is  the  relatively  low  fidelity  of  the 
timing  mechanism  where  the  worst  case  resolution  was  around 
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60ms,  and  this  was  calculated  for  a  LAN  environment.  If  the 
different  components  of  the  simulator  were  to  communicate  over 
WAN,  as  is  intended  for  future  systems,  the  dnetwork  component 
of  the  delay  could  have  been  at  the  magnitude  of  hundreds  of 
milliseconds.  Another  disadvantage  of  the  current  architecture  is 
that  it  cannot  provide  enough  data  and  would  not  provide  it  in 
such  a  way  that  the  data  could  be  used  to  capture  the  flow  of  an 
exercise  or  even  single  events  or  states  associated  with  the 
participating  entities  that  take  place  during  the  simulation.  The 
following  means  can  be  used  in  order  to  improve  the  low  fidelity 
of  the  timing: 

1 .  Events  can  be  time  stamped  at  the  originating  node  rather  then 
at  the  data  collecting  one,  this  will  eliminate  dnenvork  but  will 
require  a  time  synchronization  mechanism  between  the  nodes. 

2.  By  switching  from  Windows  to  an  operating  systems  such  as 
Linux  or  a  Real  Time  Operating  System  (RTOS)),  dtimestamp 
can  be  easily  reduced  to  less  than  1ms  because  of  the  ability 
for  user  applications  to  call  the  time  services  directly  rather 
than  by  placing  a  request  to  the  kernel.  dsampie  can  be  reduced 
if  required  but  this  will  usually  involve  hardware 
implementation  (H/D  interrupts  by  the  A/D  hardware  instead 
of  periodic  polling). 

If  there  is  a  requirement  for  capturing  the  flow  of  events  and  states 
during  a  simulated  exercise  for  the  purpose  of  review  or  replay, 
then  HLA  protocol  would  be  a  natural  candidate  for 
interconnection  between  the  different  modules  of  the  simulation. 
All  public  data  can  be  captured  by  a  data-collecting  federate  that 
is  subscribed  to  all  the  relevant  attributes  and  interactions.  For  the 
purpose  of  capturing  private  data,  the  FOM  would  have  to  be 
expended  so  that  private  data  could  be  captured  as  well  by  the  data 
collecting  federate. 


2.1.9  Integration  of  an  ERTSA-Like  System  into  the  CACR 
Simulation  Environment 

A  simulation  model  of  the  DND  Electro-optical  Reconnaissance, 
Surveillance,  Targeting  and  Acquisition  (ERSTA)  system  was  required  for 
Experiments  2  and  3  of  the  TAMSS  SA  project.  To  this  end,  a  simulation 
model  and  a  sensor  operator  station  was  modified  by  CMC  Electronics  Inc. 
and  integrated  into  the  CACR  simulation  environment  (see  Annex  A  and 
Annex  B).  The  operator  station  was  previously  developed  as  a  mock-up  to 
conduct  human  factors  experimentation  of  the  operator  station  using  an 
aircraft  FLIR  system. 
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It  is  important  to  note  that  because  the  ERTSA  system  had  has  not  been 
purchased  for  the  CH-146  fleet  within  the  timeframe  of  this  contract,  the 
ERSTA  representation  that  was  used  in  the  TAMSS  SA  project  was  intended 
only  as  generic  representation.  Accordingly,  although  an  attempt  was  made 
to  represent  the  general  functionality  of  the  anticipated  ERSTA  system,  the 
full  capabilities  of  ERSTA  system  were  not  represented  in  the  TAMSS  SA 
project.  Of  particular  note  was  that  (a)  the  range  and  fidelity  of  the  simulated 
sensor  system  (  camera  imagery)  was  approximately  40%  that  of  the 
anticipated  ERSTA  system  and  (b)  the  displays  and  controls  that  were  used  in 
this  project  were  not  selected  as  prototypes  for  the  anticipated  ERSTA 
system.  For  these  reasons,  the  ERSTA  model  that  was  used  in  the  TAMSS 
SA  project  is  heretofore  referred  to  as  the  ERTSA-like  model. 

A  description  of  the  ERTSA-like  model  operator  station  that  was  used  in 
Experiments  2  and  3  of  the  TAMSS  SA  project  is  given  in  Annex  A. 

2. 1.9. 1  Functionality  of  the  ERSTA-like  system 

The  ERSTA-like  model  that  was  used  in  the  TAMSS  SA  project 
included: 

•  A  sensor  simulation  model  to  simulate  a  Color  Day  Television 
(CDTV)  System. 

•  A  virtual  scene  display  to  simulate  real-time  video  displays  of 
sensor  imagery. 

•  A  tactical  map  display  to  show  information  received  from  the 
CACR  flight  simulator. 

•  A  communication  system  (HLA)  to  communicate  with  the 
CACR  flight  simulator. 

•  A  Centre  Console  Instrument  Display  to  show  instruments 
and  a  second  virtual  scene  (CDTV). 

•  A  tactical  display  to  show  a  second  tactical  map  for  the 
Mission  Commander. 

•  Forms  to  send  and  receive  messages. 

2. 1.9.2  Integrating  the  ERSTA-like  model  into  the  CACR 
simulation  environment 

In  the  CACR  CH-146  simulation  environment,  STAGE  software 
is  used  generate  scenarios.  For  each  entity,  such  as  a  ground 
vehicle  or  an  aircraft,  STAGE  generates  and  periodically  updates 
a  data  structure  that  contains  information  about  the  entity  state, 
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including  its  position,  speed,  orientation,  and  other  state  details. 
STAGE  provides  support  for  plug-ins,  such  that  a  user’s  custom 
software  can  access  this  data. 

In  the  TAMSS  SA  project,  a  plug-in  was  developed  that  converts 
specific  data  into  HLA  attributes  (RPR  FOM  vl.O)  that  are  then 
available  for  all  of  the  federates  that  subscribe  to  that  federation. 
The  MAK  Technologies  VR-Link  tool  is  used  as  a  wrapper 
around  the  standard  HLA  API  in  order  to  simplify  and  optimize 
the  use  of  HLA.  The  VR-Link  tool  is  used  for  functionality,  such 
as  dead  reckoning  (to  reduce  network  traffic),  smoothing 
algorithms  and  coordinate  translation.  The  ERSTA  simulation  was 
able  to  join  the  federation  and  subscribe  to  the  attributes  that 
represented  the  visual  entities  generated  by  STAGE.  The  ERSTA 
simulation  system  received  periodic  attribute  updates  for  each 
entity.  The  ERSTA  simulation  system  used  the  data  exported 
from  STAGE  to  drive  the  visual  representation  of  the  entities 
through  VEGA  software.  The  ERSTA  simulation  system  also 
received  the  Ownship  position  and  orientation  from  STAGE. 


2.2  Cognitive  Systems  Engineering  (CSE)  Framework 

A  survey  of  existing  literature  on  experimental  approaches  to  evaluating  Modelling  and 
Simulation  (M&S)  allowed  the  development  of  a  Cognitive  Systems  Engineering  (CSE) 
framework.  The  CSE  framework  combines  three  central  constructs  in  the  field  of  human- 
machine  collaboration:  situation  awareness,  workload,  and  task-relevant  performance. 
Situation  awareness  refers  to  the  operator’s  ability  to  represent  and  monitor  the  ongoing 
activity,  workload  refers  to  the  cognitive  effort  required  by  the  operator,  and  task-relevant 
performance  refers  to  the  specific  aspects  of  the  operators’  behaviour  that  relate  to  the 
machine  being  evaluated.  All  three  aspects  of  the  human-machine  system  can  be  evaluated 
objectively  and  subjectively  (the  latter  from  the  perspective  of  the  operator). 

The  CSE  framework  was  used  to  guide  the  investigations  in  the  three  TAMSS  SA 
experiments.  The  overwhelming  conclusion  is  that  the  CSE  framework  provides  a  useful  and 
coherent  approach  to  understanding  and  measuring  the  effects  of  specific  technology  on  the 
operator-machine  system. 

2.2.1  Description  of  the  CSE  Framework 

The  CSE  framework  is  based  on  the  application  of  a  human  factors  approach 
to  understanding  cockpit  design.  In  the  CSE  framework,  it  is  assumed  that 
the  strengths  and  limitations  of  the  human  operator  must  play  a  role  in 
guiding  the  development  of  new  aviation  systems.  Thus,  the  CSE  framework 
incoiporates  an  understanding  of  human  cognition  into  a  working  blueprint 
for  the  design  of  evaluation  experiments  to  support  modeling  and  simulation 
in  acquisition.  A  more  detailed  description  of  the  CSE  framework  may  be 
found  in  Annex  D  (see  also  Annex  C). 
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As  shown  in  Figure  7,  the  CSE  framework  includes  a  theoretical  construct, 
the  dynamic  mental  model,  and  three  empirical  (i.e.,  measurable)  constructs; 
situation  awareness,  workload,  and  task-relevant  performance.  A  review  of 
the  literature  indicated  that  these  three  empirical  constructs  capture  a  large 
amount  of  the  variance  in  the  human-machine  interface.  The  proximal  goal 
of  the  CSE  framework  is  to  provide  a  context  within  which  to  develop  and 
interpret  the  dependent  variables  that  are  assessed  in  an  M&S  evaluation. 
The  CSE  framework  is  not  intended  to  represent  a  complete  model  either  of 
the  human  operator  or  of  the  situation,  although  further  developments  of  the 
framework  could  expand  the  theoretical  and  predictive  power  of  the  model. 


Figure  7  -  Cognitive  Systems  Engineering  (CSE)  Framework 

The  central  theoretical  construct  in  the  CSE  framework  is  the  dynamic  mental 
model.  It  captures  the  notion  that  the  human  operator  creates  and  maintains 
an  internal  representation  of  the  ongoing  situation.  When  experimental 
methods  are  used  to  measure  performance  in  an  M&S  evaluation,  all  of  the 
measurements  are  inferences  about  the  operator’s  dynamic  mental  model. 
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The  three  empirical  constructs,  situation  awareness,  workload,  and  task¬ 
relevant  performance,  are  assumed  to  provide  a  comprehensive  (although  not 
exhaustive)  assessment  of  the  dynamic  mental  model.  Situation  awareness, 
defined  simply  as  “knowing  what  is  going  on  around  you”  (Endsley,  2000,  p. 
5)  is  a  construct  that  was  proposed  originally  to  capture  why  some  fighter 
pilots  were  more  successful  (and  therefore  lived  longer)  than  others  (Spick, 
1988).  As  discussed  by  Endsley  (2000),  SA  is  closely  tied  to  knowing  how  to 
distinguish  important  information  in  the  environment  from  less  important 
information  (selective  attention),  as  well  as  the  ability  to  quickly  comprehend 
the  importance  of  changes  to  elements  in  the  environment. 

The  second  empirical  construct  that  forms  the  core  of  the  CSE  framework  is 
workload.  Workload  is  a  familiar  construct  in  aviation  and  has  been  studied 
more  thoroughly  than  SA.  Essentially,  workload  refers  to  the  fact  that 
humans  are  limited  in  their  ability  to  process  information  and  to  respond 
appropriately.  Workload  has  proven  to  be  a  very  useful  construct  for 
understanding  changes  in  operator  behaviour  under  different  situational 
demands  and  constraints.  Technology  “improvements”  should  hypothetically 
decrease  workload,  but  in  practice,  a  technology  that  adds  information  to  the 
pilot’s  environment  and/or  requires  the  operator  to  perform  additional  tasks  is 
more  likely  to  increase  workload.  Thus,  measuring  changes  in  workload  as  a 
function  of  technology  change  is  crucial  to  understanding  how  that 
technology  influences  the  operator. 

The  third  empirical  construct  in  the  framework,  task-relevant  performance, 
refers  to  the  actions  of  the  pilot  (in  relation  to  mission  demands)  that  are 
potentially  affected  by  the  new  technology.  In  essence,  a  new  technology  is 
expected  to  change  some  aspect  of  what  the  operator  knows  and  that 
knowledge  will  be  reflected  in  his  or  her  behaviour.  Many  other  aspects  of 
the  operator’s  actions  or  behaviour  might  not  change,  however.  The  task¬ 
relevant  performance  that  is  relevant  to  any  particular  technological  change 
will  depend  on  what  that  technology  was  expected  to  influence.  For  example, 
the  addition  of  a  new  sensor  display  to  the  CEE  146  Griffon  cockpit  that 
assists  the  aircrew  in  detecting  and  identifying  targets  will  likely  also  result  in 
the  aircrew  spending  time  looking  and  interacting  with  that  display. 
Concomitantly,  the  aircrew  may  spend  less  time  using  other  sources  of 
information,  or  may  use  that  information  differently.  Thus,  defining  and 
measuring  task-relevant  performance  is  an  important  aspect  of  understanding 
the  impact  of  a  new  technology  on  performance  in  the  cockpit. 

2.2.2  Applying  the  CSE  framework 

In  the  CSE  framework,  the  three  experimental  constructs  (SA,  workload, 
task-relevant  performance)  are  second-order  reflections  of  the  pilot’s  dynamic 
mental  model  of  the  situation.  It  is  impossible  to  directly  measure  the 
dynamic  mental  model  (as  it  is  not  possible  to  directly  measure  “memory”  or 
“thinking”),  and  thus  all  measures  are  behavioural  in  the  sense  that  the 
operator  or  crew  must  elicit  some  behaviour  or  perform  some  action  that  is 
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then  assessed.  For  example,  a  subjective  assessment  of  SA  requires  that  the 
pilot  make  a  judgment  or  provide  an  evaluation.  Head  position  could  be  used 
as  an  index  of  where  the  pilot  is  attending.  Flight  path  could  be  used  as  an 
index  of  the  pilot’s  adherence  to  the  boundaries  of  a  safe  air  corridor. 

A  central  tenet  of  the  CSE  framework  is  that  human-machine  interactions  are 
often  too  complex  for  a  single  concept  to  provide  sufficient  information  to 
evaluate  the  impact  of  an  interface  on  the  operator’s  overall  behaviour.  By 
distinguishing  among  SA,  workload,  and  performance  and  the  underlying 
mental  representation  (the  dynamic  mental  model),  researchers  can  more 
clearly  operationalize  the  concepts  for  empirical  purposes.  Indeed,  due  to  the 
complexity  and  multi-dimensional  nature  of  the  dynamic  mental  model,  any 
single  construct  or  any  single  measure  of  a  construct  is  unlikely  to  capture 
sufficient  information  about  the  impact  of  a  new  technology.  In  addition, 
under  some  conditions  a  high  correlation  between  the  measures  of  the 
empirical  constructs  should  not  be  expected.  For  example,  good  SA  does  not 
always  lead  to  good  performance  and  high  workload  does  not  always  predict 
poor  performance.  Nevertheless,  these  constructs  are  related  and  in  many 
situations,  good  SA  will  predict  good  performance.  Accordingly,  it  is 
recommended  that  a  CSE  evaluation  should  include  at  least  one,  and 
preferably  multiple  measures  of  each  of  situation  awareness,  workload,  and 
task-relevant  performance.  Use  of  multiple  measures  will  allow  for  a  richer 
and  more  accurate  index  of  how  new  technologies  affect  the  human- machine 
interaction. 


2.3  TAMSS  SA  Experiment  One 

2.3.1  SA,  Workload,  and  Performance  in  Simulated  Flight:  HUD 
vs.  HDD 

Experiment  1  was  designed  to  (a)  provide  a  preliminary  evaluation  of  the 
Cognitive  System  Engineering  (CSE)  framework  and  (b)  develop  and  test  the 
technical  and  experimental  capabilities  of  the  Carleton  University  simulator 
facility.  In  this  experiment,  trained  CF  pilots  flew  a  series  of  simplified 
recce-type  missions  while  wearing  a  Helmet  Mounted  Display  (HMD).  Two 
conditions  were  compared:  Heads-Up  Display  (HUD)  versus  Heads-Down 
Display  (HDD).  In  the  HUD  condition,  the  HMD  was  equipped  with  HUD 
symbology  showing  primary  flight,  power,  and  navigation  information.  The 
HUD  symbology  was  derived  from  the  CH-146  LATEF  II  HUD.  In  the  HDD 
condition,  the  HMD  was  not  equipped  with  HUD  symbology.  Instead,  pilots 
were  required  to  look  under  the  HMD  to  acquire  the  requisite  information 
from  the  head-down  instrument  panel.  The  HDD  condition  is  similar  to  that 
experienced  by  CH-146  pilots  using  Night  Vision  Goggles  (NVGs):  when  the 
NVGs  are  not  equipped  with  a  HUD,  the  pilots  must  look  under  the  goggles 
to  read  information  from  the  instrument  panel. 
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Pilots  flew  a  series  of  the  simplified  recce-type  missions.  On  each  mission, 
they  initially  took  off  from  a  base,  located  centrally  in  the  area  of  interest. 
They  were  then  directed  (using  radio  comms)  to  find  a  series  of  waypoints 
(towers  placed  in  the  terrain)  by  an  experimenter  who  gave  them  heading 
values.  Pilots  were  given  specific  altitude  and  airspeed  parameters  that  were 
to  be  maintained  throughout  the  missions.  In  addition,  pilots  were  instructed 
to  provide  reports  (sitreps)  of  any  and  all  activity  (in  the  air  or  on  the  ground). 
Accordingly,  each  scenario  was  populated  with  a  variety  of  objects.  The 
objects  included  (a)  two  moving  formations  of  three  armored  ground  cars,  (b) 
three  stationary  pieces  of  artillery  (Howitzer  guns),  (c)  four  grounded  CHI 49 
Cormorant  helicopters,  (d)  one  wrecked  CHI 49,  (e)  two  CHI 49s  flying  in 
small  loop  formations,  (f)  two  hovering  CH-146  Griffon  helicopters,  (g)  one 
formation  of  four  CF18s  flying  in  a  wide  formation  across  a  large  portion  of 
the  terrain,  and  (h)  one  Cl 30  Hercules  fixed-wing  transport  aircraft  flying  a 
slow,  elongated  loop  pattern  that  cut  across  the  whole  width  of  the  database 
terrain,  roughly  five  kilometres  from  the  southern  edge  of  the  terrain.  All 
objects  were  placed  so  that  they  were  on,  or  intersected,  the  paths  that  pilots 
flew  in  their  missions.  Hence,  most  objects  were  close  to  the  edges  or  on  the 
diagonals  of  the  square  formed  by  the  database,  and  were  either  on  the  ground 
or  at  a  fairly  low  altitude  (below  300  feet).  The  CF18s  and  the  030  flew 
relatively  slow  and  wide  trajectories  that  intersected  the  pre-planned  mission 
routes  at  fairly  regular  intervals.  All  entities  were  scaled  to  their  normal  size 
relative  to  the  database.  The  objects  varied  in  visibility,  but  all  were  visible 
for  a  minimum  of  2  to  3  seconds.  A  more  detailed  description  of  Experiment 
One  may  be  found  in  Annex  E. 


2.3.2  CSE  Measures 

A  central  premise  in  the  CSE  framework  that  that  converging  measures 
should  be  obtained  in  order  to  gain  an  overall  perspective  that  does  not  rely 
solely  on  a  single  construct  or  single  method  of  measurement.  Accordingly, 
in  Experiment  1  all  three  dimensions  of  behaviour  outlined  in  the  CSE 
framework  were  sampled:  situation  awareness,  workload,  and  task-relevant 
performance.  Subjective  and  objective  measures  of  each  dimension  were 
developed  as  follows. 

1.  Situation  Awareness.  The  objective  index  of  situation  awareness  was  the 
percentage  of  objects  that  pilots  missed  during  each  mission.  After  each 
mission,  pilots  also  subjectively  rated  their  perceived  awareness  overall 
and  for  specific  flight  parameters  (speed,  altitude,  and  heading),  and  for 
activity  in  the  environment. 

2.  Workload.  Workload  was  assessed  through  the  presentation  of  auditory 
and  visual  probes  during  a  subset  of  the  missions.  Both  the  latency  to  the 
probes  and  the  percentage  of  probes  missed  was  measured.  Subjective 
ratings  of  workload,  globally  and  during  specific  legs  of  the  recce,  were 
also  collected. 


28 


3.  Task-relevant  Performance.  Although  performance  was  measured 
exhaustively,  the  primary  focus  in  this  experiment  was  on  deviations 
from  the  specified  airspeed  and  altitude  parameters.  In  addition, 
subjective  ratings  of  performance  and  of  task  difficulty  were  collected 
after  each  mission. 


2.3.3  Findings 

The  results  from  Experiment  1  focused  on  the  impact  of  the  HUD  versus 

HDD  on  pilot  situation  awareness,  workload  and  task-relevant  behaviour. 

2.3.3. 1  Situation  awareness 

The  results  from  Experiment  1  showed  that  the  pilots’  objective 
situation  awareness  was  worse  in  the  HUD  than  in  the  HDD 
condition.  Pilots  missed  more  objects  in  the  HUD  condition  than 
in  the  HDD  condition,  with  the  airborne  objects  (i.e.,  FI  8s, 
Hercules,  and  other  helicopters)  showing  the  greater  effect.  These 
SA  differences  between  the  HUD  vs.  HDD  conditions  do  not 
appear  to  be  due  to  relative  duration  or  relative  visibility  of  the 
various  objects,  as  there  was  a  considerable  range  for  both 
airborne  and  ground-based  objects. 

The  objective  and  subjective  measures  of  SA  showed  moderate 
convergence  in  this  experiment.  Pilots  rated  their  overall  SA  as 
somewhat  worse  in  the  HUD  than  in  the  HDD  condition,  in  accord 
with  their  actual  performance  on  the  objective  SA  (detection)  task. 
However,  the  pilots  did  not  subjectively  perceive  that  their 
performance  on  detecting  airborne  objects  as  being  worse  with  the 
HUD,  suggesting  that  their  subjective  access  to  specific  aspects  of 
their  SA  was  low. 

2. 3.3. 2  Workload 

Pilots  missed  more  of  the  auditory  and  visual  probes  (i.e.,  tones 
and  light  flashes)  when  they  were  using  HUD  symbology  than 
when  they  were  using  the  instruments.  Similarly,  their  response 
times  to  the  probes  were  also  slower  in  the  HUD  than  in  the  HDD 
condition.  These  objective  measures  show  that  pilot  workload 
higher  in  the  HUD  than  in  the  HDD  condition.  In  contrast  to  the 
objective  measure  of  workload,  pilots  subjectively  rated  their 
workload  as  similar  in  the  HUD  versus  the  HDD  condition. 
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2.3.3.3 


Task-relevant  performance/beha  viour 


Task-relevant  performance  (i.e.,  maintaining  specified  airspeed 
and  altitude)  did  not  vary  across  conditions.  Pilots  were  able  to 
maintain  the  specified  flight  parameters  within  reasonable 
boundaries  in  both  HUD  and  HDD  conditions.  Interestingly, , 
pilots  rated  their  performance  as  worse  with  the  HUD  than  with 
the  instruments.  They  indicated  that  their  performance  on 
maintaining  airspeed  and  altitude  (as  well  as  cross-checking 
instruments  and  using  information  from  the  external  scene)  was 
worse  with  the  HUD  than  with  the  HDD.  This  suggests  that  pilots 
were  aware  that  they  were  less  able  to  fulfill  all  the  demands  of 
the  missions  with  the  HUD,  but  that  a  direct  question  about 
performance  was  more  sensitive  to  these  differences  than 
questions  about  SA  or  workload.  One  possible  explanation  of 
these  results  is  that  pilots  are  more  able  to  evaluate  their 
performance  (because  they  have  direct  experience  of  it)  than  to 
evaluate  workload  or  SA,  which  are  hypothetical  constructs,  that 
may  have  different  subjective  meanings  to  different  individuals. 


2.3.4  Impact 

Experiment  1  supported  the  validity  of  the  CSE  framework  for  use  in  M&S 
assessments  of  technology.  In  particular,  this  research  showed  that 
converging  objective  and  subjective  measures  of  situation  awareness, 
workload  and  performance  should  be  used  in  order  to  obtain  an  overall 
perspective  that  does  not  rely  solely  on  a  single  construct  or  single  method  of 
measurement. 

The  importance  of  obtaining  converging  measures  was  highlighted  in  the  SA 
and  workload  results.  To  wit,  while  there  was  moderate  convergence 
between  the  objective  and  subjective  measures  of  SA,  the  pilots  did  not 
subjectively  perceive  that  their  SA  for  airborne  objects  was  worse  in  the  HUD 
versus  the  HDD  condition.  A  similar  dissociation  also  occurred  for  the 
subjective  versus  objective  measures  of  workload:  whereas  the  objective 
measures  showed  that  workload  was  higher  in  the  HUD  than  in  the  HDD 
condition,  pilots  subjectively  rated  their  workload  as  similar  across  these  two 
conditions. 

Further  support  for  the  CSE  converging  measure  approach  was  found  in  a 
separate  questionnaire  focused  on  the  use  of  the  HUD.  In  this  questionnaire, 
all  of  the  pilots  subjectively  rated  the  HUD  as  much  better  than  the  HDD 
condition  for  increasing  “eyes  out”  time.  Thus,  pilots  perceived  an  advantage 
for  the  HUD  in  the  sense  that  they  felt  they  were  more  likely  to  be  looking 
out  of  the  cockpit.  However,  as  is  evident  from  the  objective  SA  data,  the 
HUD  actually  diminished  the  pilots’  ability  to  notice  objects  in  the 
environment.  Interestingly,  although  the  pilots  reported  some  specific 
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difficulties  with  the  HUD  symbology,  they  did  not  translate  this  into  an 
overall  more  negative  evaluation. 

In  sum,  the  combination  of  measures  used  in  this  study  showed  that  the 
objective  and  subjective  indices  of  these  constructs  were  not  always  perfectly 
aligned.  It  is  clear  that  the  combination  of  the  subjective  and  objective 
measures  is  important  for  understanding  the  effects  of  the  unfamiliar  HUD 
symbology  on  situation  awareness,  workload,  and  performance. 

The  Experiment  1  finding  that  the  pilots  failed  to  detect  more  ground  and 
airborne  objects  when  using  a  HUD  is  consistent  with  other  simulation-based 
research  in  the  literature.  Although  HUDs  have  been  shown  to  be  effective  in 
specific  tasks  such  as  controlling  flight  path  and  altitude  (e.g.,  see  Fadden, 
Ververs,  &  Wickens,  2001;  Martin-Emerson  &  Wickens,  1997;  Wickens  & 
Long,  1995),  there  are  a  number  of  simulator-based  studies  suggesting  that 
pilots  may  focus  or  ‘cognitively  tunnel’  their  attention  on  HUD  symbology 
(Brickner,  1989;  Fischer,  Haines,  &  Price,  1980;  Foyle,  Stanford,  &  McCann, 
1991,  Wickens  &  Long,  1995).  Cognitive  tunneling  is  believed  to  cause 
pilots  to  miss  potentially  critical  events  in  the  external  scene.  For  example, 
Fisher  et  al.  found  that  in  a  simulated  landing  task,  pilots  were  less  likely  to 
detect  a  runway  incursion  (e.g.,  a  vehicle  driven  onto  a  runway)  when  they 
were  using  a  HUD  than  when  they  were  using  a  conventional  head-down 
display.  In  the  present  research,  the  object  detection  task  was  more 
naturalistic  than  those  used  in  previous  simulator-based  studies  of  cognitive 
tunneling.  Thus,  the  present  research  findings  are  of  particular  importance  to 
the  cognitive  tunnelling  literature  as  well  as  to  programs  aimed  toward  the 
future  development  and  implementation  of  HUDs. 

2.4  TAMSS  SA  Experiment  Two 

2.4.1  The  Impact  of  an  ERSTA-Like  System  on  the  CH-146 
Mission  Commander 

Experiment  2  involved  three  major  activities.  The  first  activity  was  to 
integrate  the  DND  ERSTA-like  model  with  the  CH-146  simulator  facility  at 
the  Carleton  University  CACR.  This  activity  included: 

1 .  Modifying  the  extant  ERSTA-like  model  to  reflect  the  core  mapping  and 
sensor  capabilities  of  the  ERSTA  system  that  is  anticipated  for  the  CH- 
146  Griffon, 

2.  Designing  and  implementing  moving  map  and  sensor  display  interfaces 
for  the  cockpit, 

3.  Making  the  ERSTA  model  compliant  with  High-Level  Architecture 
(HLA)  specifications,  and 
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4.  Linking  the  HLA-compliant  ERSTA  model  with  the  CACR  CH-146 
simulator. 

The  second  major  activity  was  to  further  develop  the  data  collection 
capabilities  of  the  simulation  environment.  The  simulation  environment  was 
significantly  more  complex  given  the  requirement  to  integrate  the  ERSTA 
simulation  with  the  flight  simulator  using  LILA. 

The  third  major  activity  in  Experiment  2  was  to  conduct  a  study  to  exercise 
the  Cognitive  Systems  Engineering  (CSE)  framework  that  has  been  proposed 
by  the  Carleton  University  CACR.  In  Experiment  2  this  was  accomplished 
by  evaluating  the  impact  of  the  prototyped  ERSTA-like  moving  map  and 
sensor  capability  on  CEL  146  Griffon  aircrew.  The  particular  focus  of  the 
experiment  was  on  how  the  ERSTA-like  capability  affects  the  situation 
awareness,  workload  and  performance  of  the  CEL  146  Mission  Commander 
(MC). 

In  this  experiment,  aircrew  consisting  of  a  Flying  Pilot  (FP)  and  a  Mission 
Commander  (MC)  completed  a  series  of  zone  recce  missions.  An  example  of 
a  mission  scenario  is  given  in  Annex  F:  further  details  regarding  the  missions 
can  be  accessed  in  the  TAMSS  SA  report  on  Experiment  2. 

Of  primary  interest  was  how  an  ERSTA-like  tactical  moving  map  and  sensor 
capability  affected  the  situation  awareness,  workload  and  performance  of  the 
CEL  146  MC  while  completing  these  missions.  Three  conditions  were 
compared  in  the  experiment: 

1 .  Paper  Map  ( P-Map ).  This  is  a  baseline  condition  that  reflects  the  current 
situation  in  the  CEL  146  where  aircrew  (i.e.,  the  MC)  navigate  using  a 
hand-held  paper  map  and  detect  and  identify  targets  without  aid  of  a 
sensor. 

2.  Moving  Map  (M-Map).  In  this  condition,  the  MC  was  provided  with  a 
digital  moving  map  (positioned  on  the  lap).  A  paper  map  was  also 
provided  for  use  at  the  discretion  of  the  MC.  As  in  the  paper  map 
condition,  the  aircrew  were  required  to  detect  and  identify  targets  without 
aid  of  a  sensor. 

3.  Moving  Map  plus  Sensor  (M-Map/Sensor).  In  this  condition,  the  digital 
moving  map  (and  the  paper  map)  and  the  ERSTA  sensor  capability  were 
provided.  The  ERSTA  sensor  (camera)  image  was  displayed  on  the  front 
centre  console,  i.e.,  where  the  current  CEL146  FLIR  image  is  normally 
located.  In  this  condition,  aircrew  were  able  to  use  the  sensor  image  to 
support  target  detection  and  identification. 

A  more  detailed  description  of  Experiment  Two  may  be  found  in  Annex  F. 
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2.4.2  CSE  Measures 


Experiment  2  was  designed  to  sample  all  three  dimensions  of  behaviour 
outlined  in  the  CSE  framework:  situation  awareness,  workload  and  task¬ 
relevant  performance.  Subjective  and  objective  measures  of  situation 
awareness  and  task-relevant  performance  were  obtained.  For  workload,  only 
subjective  measures  were  obtained. 

1.  Situation  Awareness.  Situation  awareness  was  objectively  measured  as 
the  percentage  of  relevant  objects  that  aircrew  missed  during  each 
mission.  Subjective  ratings  of  SA  were  obtained  in  questionnaires  that 
were  administered  following  each  mission. 

2.  Workload.  Workload  was  assessed  subjectively  using  questionnaires 
based  on  a  modified  NASA  TLX.  Subjective  ratings  for  global  workload 
were  obtained  as  were  ratings  for  specific  segments  (e.g.,  ingress,  recce- 
zone,  egress)  in  the  missions.  Objective  measures  of  workload  were  not 
directly  obtained. 

3.  Task-relevant  Performance.  Three  objective  measures  of  task-relevant 
performance  were  planned,  (a)  The  impact  of  the  digital  moving-map 
capability  on  navigation,  the  positioning  of  the  ownship  relative  to  the 
defined  flight  ingress  corridors  leading  to  the  RP  was  measured,  (a)  The 
impact  of  the  ERSTA-like  camera  sensor  was  objectively  defined  as  the 
distance  at  which  the  aircrew  detected  and  identified  targets,  (c)  It  was 
hypothesized  that  the  ERSTA-like  digital  moving  map  and  sensor  would 
affect  how  much  time  the  MC  spent  looking  down  and  inside  the  cockpit. 
This  was  objectively  assessed  by  recording  the  head  positioning  of  the 
MC  throughout  the  missions.  Subjective  ratings  of  performance  were 
collected  after  each  mission. 


2.4.3  Findings 

A  major  challenge  in  Experiment  2  was  to  develop  and  integrate  the  ERSTA- 
like  simulation  model  into  the  CACR  CH-146  simulation  environment  using 
HLA.  This  integration  was  successful  insofar  as  the  ERSTA-like  simulation 
was  functional  throughout  the  experiment.  However,  the  increased 
complexity  of  integrating  ERSTA-like  system  into  the  distributed  simulation 
environment  raised  a  number  of  technical  challenges.  In  particular,  online 
data  collection  was  compromised  in  Experiment  2.  These  technical 
challenges  were  identified  and  addressed  in  Experiment  3. 

The  primary  focus  in  Experiment  2  was  to  evaluate  how  the  ERSTA-like 
tactical  moving  map  and  sensor  capability  affected  the  situation  awareness, 
workload  and  performance  of  the  CH-146  MC. 
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2.4.3. 1  Situation  awareness 


Providing  the  ERSTA  moving  map  display  and  the  sensor  display 
(M-Map/Sensor  condition)  resulted  in  higher  SA  for  the  CH-146 
MC.  Objectively,  the  MCs’  situation  awareness  for  relevant 
airborne  and  ground  vehicles  was  very  high  in  that  virtually  all  of 
these  objects  were  in  detected  and  reported  in  an  appropriate  and 
timely  fashion.  MCs’  subjective  ratings  of  their  SA  for  tactical 
information  relevant  to  the  mission  were  generally  higher  in  the 
M-Map/Sensor  condition  than  in  the  P-Map  and  M-Map 
conditions.  In  addition,  the  MCs  rated  their  SA  as  higher  in  the 
M-Map/Sensor  condition  for  tracking  the  unfolding  of  a  mission 
and  for  anticipating  future  events.  Ratings  of  spatial/navigational 
awareness  in  the  MC  position  were  also  highest  in  the  M- 
Map/Sensor  condition.  Importantly,  these  ratings  showed  a  clear 
advantage  of  the  M-Map/Sensor  condition  for  locating  ownship 
relative  to  the  objective  (e.g.,  bridge)  and  relative  to  enemy 
activity  as  well  as  for  awareness  of  the  general  layout  of  the 
navigated  area. 

2.4.3.2  Workload 

Objective  measures  of  workload  were  not  obtained  in  Experiment 
2.  Subjective  ratings  of  workload  did  not  differ  dramatically 
across  the  three  conditions,  but  on  average  providing  a  moving 
map  lowered  the  MC’s  perceived  workload.  As  would  be 
expected,  MCs  rated  workload  as  being  highest  for  activity  in  the 
recce  zone  as  compared  to  the  ingress  and  egress  activities. 
Written  comments  from  participants  confirmed  that  workload  for 
the  MC  was  high  in  the  recce  zone  “due  to  the  number  of  agencies 
that  needed  to  be  contacted  on  different  frequencies”.  It  was  also 
noted  that  high  workload  for  the  MC  in  the  recce  zone  was  mainly 
associated  with  trying  to  maintain  SA  of  the  ownship  location.  It 
was  noted  that  the  digital  moving  map  reduces  workload  related  to 
navigation  thereby  freeing  more  other  tasks  (comms,  search  etc.). 

2.4. 3. 3  Task-relevant  performance/behaviour 

Head  position  data  of  the  MCs  was  collected  throughout  each 
mission.  However,  technical  difficulties  were  such  that  stable  and 
complete  data  was  only  obtained  for  one  participant.  This 
participant’s  data  showed  that  percent  head-up  time  was  greater  in 
the  M-Map  (49%)  and  M-Map/Sensor  (48%)  conditions  than  in 
the  P-Map  (33%)  condition.  This  finding  suggests  that  the 
ERSTA  capability  had  the  positive  benefit  of  allowing  MCs  to 
spend  more  time  looking  outside  the  cockpit.  Head-up  time 
should  impact  on  flight  safety  and  enhance  the  contribution  of  the 
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MC  in  detecting  and  responding  to  information  external  to  the 
cockpit.  Subjective  ratings  of  performance  for  various  tasks  in  the 
MC  position  increased  from  an  average  of  “adequate”  in  the  P- 
Map  condition  to  “good”  in  the  M-Map  and  the  M-Map/Sensor 
conditions.  As  expected,  performance  ratings  in  the  M-Map  and 
M-Map/Sensor  conditions  were  noticeably  higher  than  the  P-Map 
condition  for  the  navigation  tasks  such  as  finding  waypoints, 
reading  the  map  and  using  the  map  to  navigate.  The  ERSTA 
capabilities  were  also  rated  as  enhancing  the  MC’s  positioning  of 
the  aircraft  in  the  recce  zone. 

2.4. 3. 4  Impact 

The  results  of  Experiment  2  show  that  the  expert  participants 
perceived  an  advantage  for  the  ERSTA-like  digital  moving-map 
and  sensor  capabilities  in  their  mission  activity.  Participants 
agreed  that  the  moving  map  and  sensor  enhanced  the  MC’s 
performance  and  SA  while  generally  lowering  task  difficulty  and 
workload.  There  was  also  some  indication  of  these  benefits  being 
transferred  to  the  FP,  particularly  in  terms  of  the  aircrew’s  ability 
to  position  the  aircraft  and  to  maintain  tactical  flight.  In  addition, 
although  limited  to  one  participant,  the  head  tracking  data  showed 
that  the  MC  was  able  to  spend  more  time  looking  up  and  out  of 
the  cockpit  when  provided  with  the  digital  moving  map  and  sensor 
image. 

Experiment  2  provided  a  solid  foundation  for  developing 
Experiment  3  of  the  TAMSS  SA  project.  Of  importance  is  that  (a) 
the  ERSTA-like  system  was  effectively  modeled  and  integrated 
into  the  simulator  environment  using  HLA  protocol,  (b)  the 
missions  scenarios  that  were  developed  and  implemented 
represented  realistic  tactical  missions,  and  (c)  the  questionnaire 
battery  developed  for  obtaining  subjective  measures  proved  to  be 
sensitive  and  appropriate  for  indexing  and  differentiating  SA, 
workload  and  performance  across  the  experimental  conditions. 


2.5  TAMSS  SA  Experiment  Three 

2.5.1  Impact  of  a  Mission  Specialist  on  the  CH-146  Mission 
Commander 

The  primary  goal  of  Experiment  3  of  the  TAMSS  SA  project  was  to  exercise 
and  evaluate  the  Cognitive  System  Engineering  (CSE)  framework  for 
assessing  the  impact  of  novel  technology  on  CH-146  aircrew.  In  this 
experiment,  trained  CH-146  aircrew  completed  a  series  of  recce  missions.  On 
half  of  the  missions  the  crew  included  a  Flying  Pilot  (FP),  Mission 
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Commander  (MC),  and  a  Mission  Specialist  (MS).  On  the  other  half  of  the 
missions,  the  MS  was  not  included.  Of  interest  was  how  the  presence  versus 
absence  of  the  MS  affected  the  situation  awareness,  workload  and 
performance  of  the  CH-146  MC  while  completing  the  recce  missions. 

There  were  three  major  activities  in  Experiment  3.  The  first  activity  was  to 
extend  the  ERSTA-like  model  and  control  capabilities  and  to  further  integrate 
this  with  the  Carleton  University  CH-146  simulator  environment.  The  second 
major  activity  was  to  provide  a  more  stable  HLA-based  distributed  simulation 
environment,  including  refinements  to  the  data  collection  capabilities  of  the 
simulator  environment.  The  third  major  activity  was  to  conduct  a  study  to 
further  test  the  CSE  framework  by  examining  the  situation  awareness, 
workload  and  performance  of  the  CH-146  Mission  Commander  (MC)  under 
conditions  where  a  Mission  Specialist  (MS)  was  present  versus  a  conditions 
were  a  Mission  Specialist  was  not  present.  A  more  detailed  description  of 
Experiment  Three  may  be  found  in  Annex  G. 


2.5.2  Extend  ERSTA-Like  Model 

The  first  major  activity  was  to  extend  the  functionality  and  control 
capabilities  of  the  ERSTA-like  simulation  beyond  those  that  were  initially 
modeled  in  Experiment  2  of  the  TAMSS  SA  project  and  to  enhance  the 
integration  of  this  model  with  the  Carleton  University  CH-146  simulation 
environment.  A  summary  of  the  ERSTA-like  system,  hardware  and  software 
architecture  is  presented  in  Annex  A. 

Extending  the  ERSTA-like  system  included  modifying  the  model  that  was 
used  in  Experiment  2  to  provide: 

1 .  Additional  functionality  to  the  digital  moving  map  display,  including  a 
military  grid  overlay  for  the  digital  map,  touch  accessible  grid  read-out 
capabilities  from  the  digital  moving  map,  and  user  options  for  using 
North-up  versus  heading-up  orientation. 

2.  Control  capability  of  the  sensor  image  for  the  MC. 

2.5.3  Enhance  stability  of  simulation  environment 

The  second  major  activity  in  Experiment  3  was  to  improve  the  stability  and 
utility  of  the  simulation  environment.  The  enhancements  to  the  simulation 
environment  included: 

1 .  Improving  the  fundamental  stability  and  performance  of  the  ERSTA-like 
model. 

2.  Stabilizing  the  flight  simulator  through  programming  upgrades  and 
modification  of  the  core  simulation  and  HLA  software. 
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3.  Further  integration  of  the  ERTSA-like  model  and  the  flight  simulator. 

4.  Improvements  to  the  data  collection  capabilities  within  the  distributed 
simulation  environment. 

2.5.4  Conduct  Experiment 

The  third  major  activity  was  to  design,  conduct  and  analyze  the  experiment. 
The  primary  goal  of  Experiment  3  was  to  determine  whether  the  CSE 
framework  could  be  used  to  measure  the  impact  of  the  ERSTA-like  system  on 
the  CH-146  aircrew,  and  in  particular,  on  the  SA,  workload  and  performance 
of  the  CH-146  MC.  To  do  this  the  following  two  conditions  were  compared: 

1 .  Mission  Specialist  Present.  In  this  condition,  the  crew  included  a 
Mission  Commander  (MC),  Flying  Pilot  (FP),  and  a  Mission  Specialist 
(MS).  The  MS  assumed  primary  operation  of  the  ERSTA-like  system,  in 
and  particular,  the  sensor.  The  MC  was  able  to  view  and  interact  with  the 
digital  moving  map  and  if  desired,  take  control  of  the  sensor. 

2.  No  Mission  Specialist  Present.  In  this  condition,  the  crew  consisted  of 
the  MC  and  FP.  A  MS  was  not  present.  In  this  condition,  the  MC 
assumed  responsibility  for  operating  the  ERSTA-like  system. 

The  execution  of  Experiment  3  was  enabled  by  the  major  engineering 
activities  described  above,  as  well  as  by  the  following  activities: 

1.  Input  from  Subject  Matter  Experts  (SME)  regarding  the  functionality  and 
use  of  the  ERSTA-like  system  as  well  as  how  mission  specialists  could  be 
integrated  into  the  CH-146  aircrew. 

2.  Modification  of  the  tactical  scenarios  that  were  used  in  Experiment  2  of 
the  TAMSS  SA  project  in  order  to  provide  Fire  Mission  Support  (FMS) 
capabilities  in  the  scenarios. 

3.  Development  of  tactical  knowledge  and  the  expertise  to  allow  for 
dynamic  control  of  elements  by  the  experimenters  during  the  missions, 
including  the  escalation  of  enemy  activity. 

4.  Modifications  of  the  questionnaire  battery  from  Experiment  2  that  were 
used  for  obtaining  subjective  ratings  of  performance,  situation  awareness, 
and  workload. 

2.5.5  CSE  Measures 

In  Experiments  1  and  2  of  the  TAMSS  SA  project,  it  was  demonstrated  that 
expert’s  self  ratings  of  their  SA,  workload  and  performance  can  provide  a 
reasonable  index  concerning  the  impact  of  a  new  cockpit  technology  (i.e., 


37 


HUD).  In  addition.  Experiment  1  demonstrated  that  SA  could  be  objectively 
assessed  by  measuring  a  pilot’s  ability  to  detect  and  report  airborne  (e.g., 
other  aircraft)  and  ground  entities  (e.g.,  tanks,  downed  aircraft)  while 
performing  a  mission.  An  important  finding  from  Experiment  1  was  that  this 
objective  measure  revealed  significant  differences  in  SA  even  under 
conditions  where  the  pilots’  subjective  ratings  of  SA  were  not  different. 

In  the  present  experiment,  all  three  dimensions  of  behaviour  outlined  in  the 
CSE  framework  were  measured  subjectively:  SA,  workload,  and 
performance.  In  addition,  a  focus  was  placed  on  obtaining  an  objective 
measure  of  workload  as  well  as  objective  measures  of 
performance/behaviour. 

1.  Situation  Awareness.  Situation  awareness  was  measured  subjectively  in 
this  experiment  by  having  the  participants  complete  Likert-scale  ratings 
of  SA  after  each  mission.  Objective  measures  of  SA  were  not  obtained. 

2.  Workload.  Following  each  mission,  the  FP  and  MC  completed  separate 
Likert-scale  questionnaires  of  workload  as  well  as  workload  ratings  based 
on  a  modified  NASA  TLX.  Subjective  ratings  for  global  workload  were 
obtained  as  were  ratings  for  specific  segments  (e.g.,  ingress,  recce-zone, 
egress)  in  the  missions.  Workload  was  objectively  assessed  using  a 
visual  detection  task  whereby  the  MC  was  required  to  indicate  when  they 
detected  a  visual  target  (a  briefly  displayed  green  circle)  on  the  front 
screen.  The  targets  subtended  approximately  2  deg  of  visual  angle  and 
were  presented  every  1 5  sec  (+/-  3  sec  randomly  determined)  throughout 
the  workload  missions. 

3.  Performance/Behaviour.  Following  each  mission,  the  participants 
completed  subjective  ratings  of  their  performance  in  the  mission.  Two 
objectives  measures  were  taken.  One  objective  measure  was  the  head¬ 
positioning  of  the  MC.  It  was  hypothesized  that  when  a  MS  was  included 
in  the  crew,  the  MS  would  be  given  primary  responsibility  for  operating 
the  ERSTA-like  sensor.  For  missions  where  the  crew  did  not  include  a 
MS,  the  MC  was  required  to  control  the  sensor  image.  It  was  predicted, 
therefore,  that  MCs  would  spend  less  time  with  their  heads  down  and 
inside  the  cockpit  when  a  MS  was  present  as  compared  to  when  a  MS 
was  not  present.  The  second  objective  measure  was  the  amount  of  time 
the  sensor  was  used  throughout  a  mission.  It  was  predicted  that  the 
crew’s  use  of  the  sensor  would  be  greater  when  a  MS  was  present.  When 
a  MS  was  not  present,  the  MC  would  have  limited  time  available  for 
controlling  the  sensor. 


2.5.6  Findings 

As  in  Experiments  1  and  2,  the  primary  focus  of  Experiment  3  was  on  the 
MC.  As  summarized  below,  the  presence  of  a  MS  had  a  significant  impact  on 
the  SA,  workload  and  performance  of  the  MC.  There  were,  however, 
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indications  that  the  presence  of  the  MS  also  had  a  positive  impact  on  the 
flying  pilot.  In  particular,  flying  pilots  found  that  the  ERSTA-related 
information  was  more  useful  when  the  MS  was  operating  the  ERSTA  system 
than  then  the  MC  was  operating  the  ERSTA  system.  Related  to  this  was  a 
trend  for  higher  subjective  ratings  of  SA  related  to  tactical  awareness,  and  in 
particular  for  the  flying  pilots’  (rated)  ability  to  anticipate  future 
developments.  The  remaining  discussion  is  centered  on  the  MC. 

2.5.6. 1  Situation  A  wareness 

Subjective  ratings  showed  that  the  MCs’  self-rated  SA  ranged 
from  “moderate  to  good”,  with  SA  was  rated  as  being 
significantly  higher  in  the  MS-present  than  in  the  No-MS 
condition.  This  was  true  for  the  MCs’  ratings  of  tactical 
awareness,  spatial  awareness,  and  crew  awareness. 

2.5. 6.2  Workload 

Subjective  ratings  for  the  MC  showed  that  workload  was  rated  as 
lowest  during  ingress  and  egress  and  highest  during  activity 
associated  with  observing  targets  and  activity  in  the  recce  zone. 
During  ingress,  egress  and  general  transit  segments  of  the  mission, 
there  were  no  significant  differences  in  the  MCs’  rated  workload 
between  the  No-MS  versus  the  MS-Present  conditions.  However, 
while  in  the  recce  zone  and  also  when  observing  targets  (e.g.,  a 
choke  point  or  enemy  activity)  rated  workload  was  generally 
higher  in  the  No-MS  than  the  MS-present  condition. 

The  MCs’  workload  was  objectively  assessed  using  a  visual 
detection  task  whereby  the  MC  was  required  to  indicate  when  they 
detected  a  visual  target  (a  briefly  displayed  green  circle)  on  the 
front  screen.  The  MC  responses  to  the  visual  targets  were  divided 
into  transit  versus  observation/contact  segments.  The  transit 
segment  category  included  the  MC  responses  to  the  visual  targets 
when  the  crew  was  engaged  in  the  initial  ingress,  transit  from  the 
RP  to  the  first  observation  point,  moving  from  one  observation 
point  to  another  observation  point,  and  egress.  The 
observation/contact  category  refers  to  MC  responses  to  the  visual 
targets  when  the  crew  was  observing  a  target/objective,  submitting 
a  contact  report,  or  performing  a  FSM.  The  results  showed  that 
MCs  detected  most  (average  of  82%)  of  the  visual  targets  while  in 
transit.  Performance  while  in  transit  did  not  differ  between  the 
MS-Present  versus  the  No-MS  conditions.  In  contrast,  the  MCs 
detected  fewer  visual  targets  (average  of  56%)  while  in  an 
observation/contact  phase.  Moreover,  significantly  fewer  visual 
targets  were  detected  in  the  No-MS  than  in  the  MS-Present 
condition.  This  shows  that  the  MC  had  less  visual  attention  to 
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allocate  to  the  target  detection  task  in  the  No-MS  condition  than  in 
the  MS-Present  condition. 


2.5. 6. 3  Task-relevant  performance/behaviour 

The  MCs’  subjective  ratings  of  performance  varied  from  “slightly 
less  than  adequate”  to  “good”,  depending  on  the  task.  On  all  but 
two  tasks  (using  comms  and  positioning  the  sensor)  the  MCs  rated 
their  performance  as  being  better  when  a  MS  was  present  as 
compared  to  when  no  MS  was  present. 

One  objective  measure  of  performance/behaviour  was  the  head¬ 
positioning  of  the  MCs.  Head  position  data  for  the  MC  was 
collected  throughout  each  mission.  Overall,  percent  head-up  time 
for  the  MC  was  significantly  greater  (better)  when  a  MS  was 
present  (37%  head-up  time)  as  compared  to  when  a  MS  was  not 
present  (20%  head-up  time).  Head-up  time  should  impact  on 
flight  safety  and  performance:  enhanced  head-up  time  should 
facilitate  the  MC’s  ability  to  detect  and  respond  to  information 
external  to  the  cockpit.  A  close  examination  of  the  data  showed 
that  when  there  was  no  MS  present,  the  MC  spent  more  time 
looking  at  the  ERSTA  sensor  image  and  the  ERSTA  digital 
moving  map.  This  extra  time  on  the  sensor  was  likely  due  to  the 
additional  requirement  on  the  MC  to  operate  the  sensor  when 
there  was  no  MS.  Operating  the  sensor  requires  frequent  use  of 
the  digital  map  orienting  and  moving  the  sensor  (touch-click 
operation).  These  findings  concurs  with  the  subjective  ratings  of 
difficulty  where  MCs  indicated  that  “use  of  the  sensor”, 
“positioning  the  sensor”  and  “getting  information  from  the  sensor” 
was  quite  difficult  in  the  No-MS  condition. 

A  second  objective  measure  of  performance/behaviour  was  sensor 
usage.  The  average  percent  time  that  the  sensor  was  used  was 
computed  relative  to  the  overall  mission  time.  When  a  MS  was 
present,  the  sensor  was  moved  by  the  MS  for  an  average  of  40% 
of  the  overall  mission  time.  When  a  MS  was  present,  the  sensor 
was  controlled  by  the  MC  only  1.7%  of  the  time:  thus,  the  MS  had 
the  primary  responsibility  for  moving  the  sensor.  When  there  was 
no  MS  present,  the  sensor  was  used  by  the  MC  for  an  average  of 
27%  of  the  overall  mission  time.  Thus,  when  there  was  no  MS, 
the  sensor  was  used  less  then  half  of  the  time  compared  to  a 
situation  where  a  MS  was  included  as  part  of  the  aircrew. 


2.5.7  Impact 

The  primary  goal  of  Experiment  3  was  to  extend  the  evaluation  of  the 
Cognitive  System  Engineering  (CSE)  framework  for  evaluating  the  impact  of 
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novel  technology  on  aircrew.  To  do  this,  the  impact  of  an  ERSTA-like 
system  was  assessed  with  a  particular  focus  on  how  ERSTA  affects  the 
situation  awareness,  workload  and  performance/behaviour  of  the  MC.  Two 
conditions  were  examined:  Mission  Specialist  Present  versus  No  Mission 
Specialist. 

The  results  of  Experiment  3  provide  a  clear  picture  of  how  measuring  the 
three  CSE  constructs  (situation  awareness,  workload,  and  task-relevant 
performance)  can  provide  a  broad  but  integrated  assessment  of  how  novel 
technology  can  impact  aircrew.  In  Experiment  3,  the  use  of  the  ERSTA-like 
system  benefited  from  the  addition  of  a  mission  specialist  to  the  CH-146 
aircrew.  In  particular,  when  a  MS  was  present  to  operate  the  ERSTA  system, 
the  MC,  freed  from  the  increased  demands  of  operating  the  sensor,  had  more 
mental  attention  to  put  towards  the  primary  demands  of  the  MC  role. 

Experiment  3  was  also  technically  progressive.  The  ERTA-like  model  was 
successfully  extended  to  provide  an  enhanced  level  of  functionality  thereby 
enabling  the  aircrew  to  use  the  digital  moving  map  and  the  sensor  capabilities 
in  a  realistic  and  appropriate  manner.  The  Carleton  University  HLA-based 
distributed  simulation  environment  was  robust  and  stable.  The  data  collection 
utility  was  stable  and  accurate  throughout  the  experiment  and  the  HLA-based 
distributed  simulation  which  the  CACR  used  to  connect  the  model  of  the 
ERSTA  system  to  the  CACR  CHI 46  flight  simulator  ran  flawlessly  for  a 
minimum  of  eight-to-ten  hours  per  day  across  twelve  days  of  testing. 

2.6  Summary  of  TAMSS  SA  Project  Activities 

Two  major  activities  were  completed  in  the  TAMSS  SA  project.  The  first  major  activity  was 
to  establish  a  research-enabling  CH-146  simulation  environment  at  the  Carleton  University 
CACR.  The  second  major  activity  was  to  conduct  three  graduate-level  research  studies  on 
Situation  Awareness  (SA).  These  major  activities  resulted  in  the  following  deliverables: 

1 .  The  development  of  a  CHI 46  Griffon  simulation  capability  at  the  Carleton 
University  Centre  for  Applied  Cognitive  Research, 

2.  The  development  of  a  Cognitive  Systems  Engineering  (CSE)  framework  to  guide 
the  simulation-based  evaluation  process, 

3.  Three  experiments  that  both  assessed  the  engineering  system  and  the  theoretical 
framework,  and 

4.  This  final  document,  which  explains  and  summarizes  the  process. 

The  Carleton  University  C  ACR  CH-146  simulation  environment  was  developed  across  the 
extent  of  the  TAMSS  SA  project.  This  cost  effective  and  malleable  environment  was  proven 
to  be  an  effective  platform  for  prototyping  and  exercising  systems  and  for  measuring  the 
impact  of  new  technologies  in  a  dynamic  simulation  environment. 
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The  CSE  framework  was  used  to  guide  simulation-based  evaluations  in  the  three  TAMSS  SA 
studies:  this  framework  provides  an  effective  structure  for  simulation-based  evaluations 
human-machine  systems.  For  each  of  the  three  TAMSS  SA  studies,  trained  CH-146  aircrews 
completed  simulated  reconnaissance  missions.  In  Experiment  1,  CH-146  pilots  showed 
reduced  situation  awareness  and  increased  workload  when  they  were  using  a  heads-up 
display.  In  Experiment  2,  ERSTA-like  sensor  capabilities,  combined  with  a  digital  moving 
map,  allowed  CH-146  mission  commander  to  gain  enhanced  situation  awareness  and  at  times 
reduced  workload.  In  Experiment  3,  it  was  shown  that  the  presence  of  a  mission  specialist  to 
operate  an  ERSTA-like  sensor  system  allowed  the  CH-146  mission  commander  to  have 
increased  heads-up  time,  substantially  decreased  workload,  and  enhanced  situation  awareness. 

The  TAMSS  SA  project  is  also  important  in  demonstrating  that  the  fidelity  obtained  with  a 
relatively  inexpensive  simulation  based  on  PC  platforms  is  sufficient  for  exercising  and 
measuring  the  impact  of  technology  in  an  operator-machine  system.  In  addition,  the  TAMSS 
SA  project  showed  how  High-Level  Architecture  (HLA)  can  be  used  to  connect  distributed 
models  in  a  manner  that  is  achievable  in  simulation-based  evaluation  programs.  In  summary, 
the  TAMSS  SA  project  provides  a  guide  for  the  implementation  of  simulation-based 
acquisition  in  a  cost  effective,  malleable  platform. 
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3.  CSE  Framework  and  Acquisition  Programs 

3.1  Role  of  the  CSE  Framework  in  Acquisition  Programs 

As  illustrated  in  Figure  8,  acquisition  programs  follow  a  sequence  of  steps  ranging  from 
Concept  Development  and  Exploration  (CDE)  through  to  Options  Analysis,  Definition  and 
Implementation.  For  each  of  these  steps  in  the  acquisition  process,  risk  can  be  reduced 
through  the  use  of  CSE-based  modelling  and  simulation  for  design,  rapid  prototyping  and 
simulation-based  evaluation  activities. 

3.1.1  An  Iterative  Modelling  and  Simulation  Process 

In  Figure  8,  the  top-level  callout  shows  three  stages  involved  in  modelling 
and  simulation:  design,  rapid  prototyping  and  SBE.  The  left-to-right 
sequence  (as  indicated  by  the  horizontal  arrows)  illustrates  that  modelling  and 
simulation  normally  starts  with  a  design  phase,  followed  by  rapid  prototyping 
and  SBE.  An  important  feature  of  modelling  and  simulation  activities, 
however,  is  that  iterative  feedback  to  earlier  activities  is  enabled.  This 
iterative  feedback  is  shown  as  the  vertical  arrows  in  the  top-level  callout  of 
Figure  8. 

The  iterative,  feedback-enabled  approach  is  required  for  modelling  and 
simulation  to  be  maximally  effective  for  acquisition.  For  example,  the 
development  of  a  rapid  prototype  and/or  the  use  of  simulation-based 
evaluation  may  reveal  system  limitations  or  raise  alternative  solutions  that 
were  not  identified  in  the  initial  design  stage.  Similarly,  while  activity  in  the 
SBE  stage  may  validate  the  rapid  prototypes,  SBE  activity  can  also  reveal  the 
need  for  changes  in  the  prototypes. 

At  a  theoretical  level,  the  feedback  between  the  stages  can  be  thought  as  a 
cascading  error-correcting  mechanism,  similar  in  principle  to  that  used  in 
Perceptual  Control  Theory  (PCT:  Hendy  et  al.,  2002).  On  this  view,  feedback 
continues  until  the  difference  between  outputs  from  the  stages  reaches  an 
acceptable  criterion.  More  generally,  this  process  can  be  viewed  as  satisfying 
a  set  of  hierarchically  defined  set  of  goals.  Top-level  goals  will  be  consistent 
across  the  stage  of  the  acquisition  process.  However,  specific  goals  will  be 
driven  by  the  concept  development,  option  analysis,  definition  and 
implementation  stages. 
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Figure  8  -  Integration  of  Modelling  and  Simulation  into  the  Acquisition  Process 

3.1.2  CSE-Based  Modelling  and  Simulation 

The  CSE  framework  that  was  developed  in  the  TAMSS  SA  project  provides  a 
structure  for  modelling  and  simulation  activities  in  each  of  design,  rapid 
prototyping  and  Simulation-Based  Evaluation  (SBE).  For  design  and  rapid 
prototyping,  the  constructs  in  the  CSE  framework  orient  activity  toward 
consideration  of  how  the  human- machine  system  will  potentially  affect 
operator  SA,  workload  and  task-relevant  performance.  The  CSE  framework, 
however,  is  particularly  germane  to  SBE:  the  CSE  framework  provides  both  a 
conceptual  structure  and  strong  methodological  guidance  for  SBE  activities. 

The  use  of  a  CSE-based  modelling  and  simulation  approach  for  acquisition 
programs  is  based  on  three  fundamental  premises: 

1.  Premise  1:  Human-machine  systems  are  constrained  by  limitations  in 
human  abilities.  Whereas  technologies  can  be  further  developed  and 
modified,  humans  remain  inherently  limited  in  their  ability  to  attend, 
process  information,  time-share  tasks,  comprehend  events,  make 
decisions  and  elicit  correct  actions.  To  some  extent,  training  can  counter 
these  limitations  and/or  allow  these  limitations  to  be  managed.  However, 
fundamental  limitations  in  human  abilities  can  not  be  eliminated.  It  is 
important  therefore,  to  fully  consider  human  abilities  and  limitations  from 
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the  early  stages  of  concept  development  through  to  system 
implementation.  For  acquisition  programs,  the  CSE  framework  provides 
a  manageable  and  comprehensive  approach  by  grouping  the  examination 
of  human  abilities  into  three  constructs:  SA,  workload  and  task-relevant 
behaviour. 

2.  Premise  2:  Systematic  assessment  and  measurement  is  required.  In 
accord  with  thte  CSE  framework,  assessing  the  capabilities  of  a  human- 
machine  system  is  achieved  through  (a)  the  systematic  manipulation  of 
relevant  variables  that  exercise  the  human-machine  system  and  (b)  the 
measurement  of  operator  SA,  workload  and  performance.  An  advantage 
of  modelling  and  simulation  is  that  it  can  be  used  to  represent  the 
intended  use,  dynamics  and  functionality  of  the  human-machine  system 
in  an  interactive  and  realistic  environment. 

3.  Premise  3:  CSE-based  modelling  and  simulation  lowers  acquisition  cost 
and  enhances  likelihood  of  success.  Information  derived  from  CSE-based 
modelling  and  simulation  can  both  validate  and  expedite  activities  in  the 
acquisition  process,  ranging  from  concept  development  through  to 
implementation.  Further,  CSE-based  modelling  and  simulation  involves 
end-users  resulting  in  better  informed  and  more  confident  procurement. 
This  empowers  procurement  personnel  through  informed  decision 
making  which  in  turn  allows  the  acquisition  process  to  proceed  in  a 
timely  and  cost-effective  manner  while  increasing  the  likelihood  of 
successfully  acquiring  the  optimal  system. 

3. 1.2. 1  CSE-based  modelling  and  simulation:  Simulation 
fidelity  issues 

Obtaining  valid  and  meaningful  measures  of  the  operators’  SA, 
workload  and  performance  may  require  special  consideration 
regarding  the  fidelity  of  the  synthetic  environment.  In  SBE  one 
important  criterion  concerning  fidelity  is  whether  the  participants 
“accept”  the  simulation  environment  as  a  meaningful 
representation.  This  criterion  essentially  reduces  to  whether  the 
operators  become  effectively  immersed  in  the  synthetic 
environment  to  the  extent  that  they  (a)  operate  the  systems  in  an 
appropriate  fashion  and  (b)  perform  the  tasks  in  a  manner 
analogous  to  how  they  would  operate  the  systems  in  a  real 
platform.  Besides  subjective  experience,  there  are  a  number  of 
components  to  a  simulation  environment  that  for  which  fidelity 
must  be  considered.  As  an  example,  in  the  TAMSS  SA  project 
fidelity  considerations  for  the  CACR  CH-146  simulation 
environment  included  the  following: 

1 .  Visual  system.  The  resolution  and  update  rate  of  the  visual 
must  be  sufficient  to  enable  operators  to  view  and  use  the  Out- 
The-Window  (OTW)  scene  effectively.  For  example,  in  the 
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TAMSS  SA  project,  the  fidelity  of  the  visuals  were  sufficient 
to  enable  pilots  to  use  the  OTW  scene  for  low-level  and  NOE 
tactical  flight. 

2.  Flight  controls.  The  cyclic  and  collective  controls  in  the 
TAMSS  CH-146  simulator  are  of  low  fidelity.  These 
controls,  however,  were  tuned  to  approximate  the  response  of 
the  CH-146  and  were  sampled  at  a  rate  that  allowed  for 
predictable  and  timely  responses.  These  controls  are  not  of 
high  enough  fidelity  to  allow  for  pilot  flight  (aviate)  training, 
but  were  sufficient  to  allow  pilots  to  fly  the  simulated  CH-146 
at  tactical  levels  (including  hover),  usually  with  less  than  5 
minutes  of  practice. 

3.  Instrumentation.  In  many  simulation  environments,  glass 
panel  (often  LCD)  instrument  displays  are  used.  These 
displays  visually  correspond  to  the  instruments  in  the  real 
aircraft,  but  do  not  provide  tactile  feedback.  In  many  cases  a 
glass  panel  representation  is  suitable.  In  other  cases,  tactile 
feedback  is  an  important  component  of  the  operator’s 
experience.  For  example,  experienced  CH-146  aircrews  rely 
upon  tactile  feedback  when  entering  information  into  the 
CDU  (button  shapes,  perceptible  clicks).  For  this  reason,  the 
TAMSS  CH-146  system  includes  CDU  bezels  and 
Communication  Selection  Control  (CSC)  panels  that  are  very 
similar  to  those  in  the  CH-146  helicopter  and  that  provide 
high  tactile  fidelity. 

4.  Scenario  generation.  To  measure  SA,  workload  and 
performance  in  a  manner  that  provides  face  and  construct 
validity,  simulation-based  evaluations  may  require 
participants  to  perform  in  scenarios  that  are  as  realistic  as 
possible.  Thus,  scenario  generation  must  allow  for  the  use  of 
accurate  models  (e.g.,  models  of  vehicles,  weapons)  and  when 
feasible,  provide  a  dynamic  flow  of  events.  Depending  on  the 
system  that  is  under  evaluation,  the  scenario  generation  utility 
may  require  the  ability  to  vary  environment  factors  such  as 
wind  and  visibility.  The  fidelity  of  the  environmental  models 
will  depend  on  the  specific  systems  and  questions  that  are 
being  examined  in  the  simulation-based  evaluation. 

5.  Data  collection.  The  fidelity  of  data  collection  in  a  simulation 
environment  is  important  and  complex.  Issues  surrounding 
data  collection  are  more  complex  when  a  distributed 
simulation  environment  is  used.  Some  variables  must  be 
sampled  at  high  rates  (e.g.,  60  Hz  or  better),  whereas  others 
can  be  sampled  at  lower  rates.  For  evaluations  using 
operators’  response  latencies  as  a  measure,  data  collection 
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may  need  to  be  accurate  to  1  -  5  milliseconds.  The  CSE 
framework  can  be  used  to  select  measures  and  to  specify  the 
level  of  resolution  that  is  required  for  these  measures  to  be 
valid  and  useful.  This  will  directly  affect  the  design  and 
system  architecture  of  the  simulation  environment  that  will  be 
used  for  the  evaluation.  A  detailed  discussion  of  data 
collection  in  the  TAMSS  SA  project  is  presented  in  Section 
2.1.8. 

A  useful  criterion  concerning  data  collection  fidelity  is  whether 
variations  in  the  key  factors  examined  in  the  simulation-based 
evaluation  resulted  in  detectable  differences  in  the  CSE 
measure(s).  In  general,  if  the  CSE  measure(s)  shows  change  that 
correspond  to  variation  in  the  manipulated  variables,  then  it  is 
concluded  the  manipulated  variables,  the  measures,  and  the 
simulation  environment  are  of  sufficient  fidelity.  However,  firm 
conclusions  are  difficult  to  make  when  the  manipulated  variables 
do  not  affect  the  measured  variables:  this  is  referred  to  as  a  “null 
effect”.  Null  effects  may  occur  because  the  manipulated  variable 
simply  does  not  have  an  impact  on  the  operator’s  SA,  workload  or 
performance.  Alternatively,  however,  null  effects  may  reflect  (a) 
weak  (low  fidelity)  or  possibly  irrelevant  manipulations,  (b)  an 
insensitive  (low  fidelity)  or  irrelevant  measured  variable,  or  (c)  an 
inadequate  (low  fidelity)  simulation  environment.  In  sum,  it  is 
generally  very  difficult  to  interpret  null  effects.  An  evaluation 
that  produces  only  null  effects  will  require  further  investigation 
into  the  fidelity  of  the  manipulated  variables,  the  measured 
variables,  and  the  simulation  environment. 

3.1.3  Reduced  Costs  and  Time-Lines  Using  Modelling  and 
Simulation 

A  fundamental  premise  underlying  the  use  of  modelling  and  simulation  in  the 
acquisition  process  is  that  there  must  be  an  allowance  for  iterative  cycles 
among  the  stages  as  shown  in  Figure  8  -  Integration  of  Modelling  and 
Simulation  into  the  Acquisition  Process.  Iteration  does  not  imply  increase 
cost  or  extended  timelines.  Time-lines  associated  with  rapid  prototyping  and 
simulation-based  acquisitions  are  relatively  short.  Importantly,  the 
information  derived  from  these  activities  can  both  validate  and  expedite 
concept  development  and  assessment/design.  These  modelling  and 
simulation  activities  will  result  in  better  informed  and  more  confident 
procurement.  This  empowers  personnel  through  informed  decision  making 
which  in  turn  allows  the  acquisition  process  to  proceed  in  a  timely  and  cost- 
effective  manner  while  increasing  the  likelihood  of  successfully  acquiring  the 
optimal  system. 
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3.1.4  Conclusions 


Humans  are  limited  in  their  ability  to  process  information.  As  such,  the 
abilities  and  limitations  of  operators  must  be  considered  whenever  a  human- 
machine  system  is  under  consideration.  The  CSE  framework  that  was 
developed  in  the  TAMSS  SA  project  provides  a  structure  for  modelling  and 
simulation  activities.  The  CSE  framework  is  central  to  SBE:  the  CSE 
framework  provides  both  a  conceptual  structure  and  strong  methodological 
guidance  for  SBE  activities.  In  particular,  the  CSE-based  modelling  and 
simulation  requires  the  use  of  an  evaluation  process  whereby  specific 
variables  are  systematically  manipulated  and  the  impact  on  the  operators’  SA, 
workload  and  performance  are  measured. 

The  TAMSS  SA  project  unequivocally  demonstrated  that  the  fidelity  of  the 
TAMSS  simulation  environment  is  sufficient  to  support  simulation-based 
evaluation  of  CH-146  technologies.  Specifically,  the  TAMSS  SA  project 
demonstrated  that  the  fidelity  obtained  with  a  relatively  inexpensive 
simulation  environment  based  on  PC  platforms  is  sufficient  for  exercising  and 
measuring  the  impact  of  technology  (e.g.,  heads-up  displays,  ERSTA-like 
sensor  capabilities)  in  the  CH-146.  The  project  also  demonstrated  that 
complex,  accurate  and  relevant  data  can  be  obtained  using  PC-based 
simulation  tools.  Finally,  the  project  showed  how  High-Level  Architecture 
(HLA)  can  be  used  to  connect  distributed  models  in  a  manner  that  is 
achievable  in  simulation-based  acquisition  programs. 
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4.  Integration  Into  Training 

Throughout  the  conduct  of  the  work  associated  with  the  establishment  of  a  CH-146  simulation 
environment  at  the  Carleton  University  CACR,  as  well  as  throughout  the  conduct  of 
experiments,  it  became  readily  apparent  that  NTS-like  devices  had  the  potential  to  serve  a  far 
broader  range  of  applications  than  those  exercised  during  the  performance  of  the  TAMSS  SA 
project.  Comments  were  received  from  industry  and  government  that  alluded  to  such 
potential;  the  most  frequent  of  which  was  associated  with  the  potential  application  of  NTS- 
like  devices  to  the  domain  of  training.  The  following  sections  describe  these  potential 
applications  to  training  in  greater  detail  and  explore  the  nature  of  modifications  to  the  NTS 
system  that  are  believed  to  be  required  in  order  to  pursue  such  applications. 

4.1.1  Training 

In  order  to  meet  the  experimental  requirements  of  the  TAMSS  SA  project  the 
NTS  system  was  outfitted  with  low-  to  medium- fidelity  representations  of 
CH-146  avionics  systems  and  flight  controls.  As  has  been  described  in  detail 
in  previous  sections  of  this  document,  the  avionics  systems  were  represented 
in  a  variety  of  fashions,  including  virtual  representations  presented  on  flat 
panel  displays  (fitted  with  touch  screens),  as  well  as  with  simulated  bezels 
and  control  panels.  The  resulting  cockpit  environment  was  described  by 
numerous  operators  and  visiting  Subject  Matter  Experts  (SMEs)  as  being 
applicable  to  training  activities  coincident  with  Part-Task  Training  (PTT), 
Cockpit  Procedures  Training  (CPT),  Tactics  Training  and  Mission  Rehearsal 
requirements. 

4. 1. 1. 1  Tactics  Training  and  Mission  Rehearsal 

Tactics  Training  and  Mission  Rehearsal  systems  must  allow 
operators  to  fly  a  simulated  CH-146  Griffon  in  a  tactically 
relevant  synthetic  environment.  The  purpose  of  a  Tactics  Trainer 
is  to  teach  a  crew  how  to  correctly  employ  their  tactical  equipment 
in  an  operationally  relevant  and  crewed  environment  (as  opposed 
to  learning  how  to  operate  the  equipment  in  an  artificially  tranquil 
environment).  As  such,  the  effective  management  of  crew 
resources  is  also  a  benefit  of  a  Tactics  Training  system.  Unlike  a 
Full-Motion  Flight  Simulator  (FMFS),  the  intent  of  a  Tactics 
Trainer  is  not  to  teach  “hands-and-feaf  ’  operation  of  the  aircraft, 
nor  is  it  intended  to  teach  emergency  procedures. 

4. 1. 1.2  Cockpit  Procedures  Training 

The  purpose  of  a  CPT  is  to  provide  a  simulated  cockpit 
environment  that  allows  a  crew  to  familiarize  themselves  with 
aircraft  systems.  Typically,  a  CPT  takes  the  form  of  a  full  scale, 
functional  replica  of  an  aircraft  cockpit.  A  CPT  should  allow 
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routine  and  critical  cockpit  procedures  to  be  learned,  thus  reducing 
the  amount  of  time  being  demanded  of  the  real  aircraft  or  FMFS. 
An  intent  of  a  CPT  is  to  teach  the  location  and  feel  of  cockpit 
controls.  As  such,  it  must  provide  a  tactile  representation  of  the 
cockpit  that  affords  an  opportunity  to  the  crew  to  physically 
interact  with  all  switches,  flight  controls  and  instruments.  Crew 
Resource  Management  (CRM)  and  cooperation  can  also  be  taught 
in  a  CPT.  The  simulated  environment  of  a  CPT  should  include  all 
visual  and  auditory  cues,  so  as  to  allow  the  crew  to  identify  cause 
and  effect  relationships  associated  with  the  operation  of  aircraft 
systems. 

4. 1.1. 3  Part-  Task  Training 

Part-Task  Trainers  focus  on  the  training  surrounding  a  specific 
aircraft  system  or  group  of  systems.  A  PTT  should  afford  an 
individual  crew  member  with  the  ability  to  interact  with  a 
simulated  system,  without  the  need  to  occupy  a  higher  fidelity 
system  (i.e.  CPT,  TT  or  FMFS).  Systems  such  as  the  Control 
Display  Unit  (CDU)  are  ideal  candidates  for  PTT  since  they  are 
complex  in  nature,  but  can  be  isolated,  to  a  degree,  from  the  rest 
of  the  cockpit.  In  certain  instances,  a  realistic  tactile  interaction 
with  a  system  of  focus  may  be  required.  However,  in  many  cases 
interaction  with  virtual  representations  of  aircraft  systems  will  be 
adequate. 

4. 1. 1.4  Current  Limitations  of  the  NTS 

The  NTS  was  found  to  have  limitations  that  would  preclude  the 
use  of  the  system,  in  its  present  state,  in  supporting  a  training 
regimen  (TT,  PTT  or  CPT).  These  limitations  are  described  as 
follows: 

1.  Level  of  Software  Maturity.  The  NTS  system  was  designed  to 
fulfill  the  needs  of  a  Technology  Demonstration  Project 
(TDP),  specifically  that  of  the  TAMSS  TDP.  The  nature  of 
the  TAMSS  TDP  resulted  in  design  goals  for  the  NTS  system 
that  were  not  intended  to  provide  scalability  and  longevity 
beyond  the  end-date  of  the  TDP.  As  such,  the  level  of 
maturity  of  much  of  the  software  that  was  developed  to 
facilitate  the  integration  of  the  COTS  components  that 
comprise  the  NTS  is  not  suitable  to  form  the  foundation  for  a 
training  system. 

2.  Stability.  Related  to  point  (1),  the  stability  requirements  of  a 
system  that  will  be  used  to  deliver  training  are  different  from 
those  used  to  support  experimentation.  For  the  purpose  of  this 
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discussion,  “stability”  is  used  to  refer  to  the  period  of  time 
over  which  a  system  can  be  observed  to  operate  correctly, 
absent  of  any  error  of  sufficient  severity  as  to  compromise  the 
validity  of  the  task  being  performed.  Stability  can  be  likened 
to  a  measure  of  Mean  Time  Between  Failures  (MTBF)  for 
simulator  systems.  A  system  that  will  be  used  to  deliver 
training  has  a  more  stringent  requirement  for  stability  than  a 
system  used  to  support  experimentation  such  as  that 
conducted  under  the  TAMSS  SA  project  due  to  the  potentially 
negative  impact  upon  training  that  could  result  from  system 
failure.  Although  the  NTS,  under  careful  supervision  of  the 
CAC-R  staff,  was  found  to  sufficiently  stable  to  conduct  SA 
experiments,  it  is  not  felt  that  the  system  is  stable  enough  in 
its  present  form  to  support  training. 

3.  Open  Architecture.  Simulation  systems,  in  general,  benefit 
from  an  open  architecture.  “Open  Architecture”  is  a  term 
used  to  describe  a  hardware  or  software  design  that  has 
published  specifications  allowing  third  parties  to  develop  add¬ 
on  modules  that  can  be  easily  incoiporated  into  a  system 
derived  from  the  design.  In  this  case,  an  open  architecture 
will  be  of  benefit  to  a  training  device  by  affording  a  flexible 
means  of  upgrading  the  device  to  match  the  evolving 
complement  of  equipment  on  the  CH-146  Griffon  helicopter. 
An  open  architecture  is  also  of  benefit  due  to  the  potential  for 
component  re-use  which  is  fostered  by  a  requirement  for  the 
development  of  modular  systems  that  adhere  to  published 
specifications.  Although  portions  of  the  NTS  feature  an  open 
architecture,  the  scope  of  the  system  that  is  “open”  is 
insufficient  to  result  in  a  significant  benefit  in  a  training 
context. 

4.  Focused  Fidelity.  The  fidelity  requirements  of  PTT- and 
CPT-level  training  devices  will  require  the  incorporation  of 
cockpit  elements  that  exceed  those  currently  available  in  the 
NTS.  These  elements,  such  as  switches  and  other  controls 
associated  with  the  fuel  management  system  are  examples  of 
“focused  fidelity”.  That  is,  the  attention  to  fidelity  is  focused 
on  those  parts  of  the  cockpit  that  are  deemed  to  be  important 
to  the  training  task,  rather  than  mandating  a  specific  level  of 
fidelity  to  be  applied  to  the  representation  of  the  entire 
cockpit.  The  NTS  features  focused  fidelity  for  pieces  of 
equipment  relevant  to  the  TAMSS  TDP  (CDU  bezels,  CSC 
panel)  but  lacks  fidelity  in  other  portions  of  the  cockpit  that 
will  be  important  to  TT-,  PTT-  and  CPT-level  training. 

5.  Fidelity  of  Models  of  Aircraft  Systems.  Although  the  NTS 
cockpit  environment  looks  and  feels  like  a  CH-146  Griffon  in 
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many  respects,  the  models  of  aircraft  systems  that  underlie  the 
virtual  instruments  are  generic  in  nature.  The  NTS 
representations  of  the  fuel  system,  electrical  system,  hydraulic 
system  and  engines  are  built  upon  representations  available  in 
the  commercial  HEL1SIM  product,  and  were  not  developed 
using  authoritative  data  for  the  Bell  412.  In  order  to  teach 
proper  cause  and  effect  relationships  between  these  and  other 
systems,  a  higher  fidelity  representation  of  these  systems  must 
be  developed  or  integrated. 


4.1.2  Summary 

Significant  modifications  would  be  required  in  order  to  prepare  the  NTS  for 
use  in  a  training  regimen.  However,  the  potential  role  for  NTS-like  devices  in 
a  vertically  integrated  training  program  should  not  be  underestimated.  SMEs 
who  participated  in  TAMSS  SA  experiments  expressed  unsolicited  support 
for  the  notion  that  upgraded  NTS-like  simulators  could  directly  support 
tactics  training,  mission  rehearsal  and  doctrine  development.  One  SME  noted: 

“Quantum  mission  improvements  were  obvious  from  one  mission  simulation 
to  the  next,  and  we  never  turned  a  real  rotor.  ” 

The  consideration  of  applicability  to  PTT  and  CPT  tasks  were  made  in  order 
to  explore  the  applicability  of  an  NTS-like  device  to  a  vertically  integrated 
training  solution,  such  as  that  represented  by  the  Integrated  Simulation 
Training  System  (ISTS)  concept  currently  circulating  within  the  Canadian 
Forces. 
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1  OVERVIEW 

1.1  Modelling  and  Simulation  Objective 

The  objective  of  the  rapid  prototype  is  to  provide  an  Electro-optical  Reconnaissance, 
Surveillance,  Targeting  and  Acquisition  (ERSTA)  Human  Factors  Engineering  (HFE)  mock-up 
to  support  Situational  Awareness  (SA)  studies  conducted  by  the  Carleton  University  Centre  for 
Applied  Cognitive  Research  (CACR)  for  the  Tactical  Aviation  Mission  System  Simulation 
(TAMSS)  SA  Technology  Demonstration  Project  (TDP).  The  Situational  Awareness  Studies  are 
an  important  component  of  TDP  as  they  will  help  to  formalize  the  requirements  capture  process 
essential  to  Simulation  Based  Acquisition  (SBA).  TAMSS  is  an  approved  Defence  R&D 
Technology  Demonstration  Project. 

The  Air  Force  requires  improved  Modelling  and  Simulation  (M&S)  capability  to 
support  SBA,  mission  rehearsal,  human  factor  assessments  and  training.  The  TAMSS  project 
will  focus  on  M&S  for  Acquisition.  TAMSS  will  use  distributed  and  local  M&S  techniques  to 
link  high  fidelity  Defence  Research  Establishment  (DRE)  component  models  with  CHI 46  crew 
station  simulators  for  system  and  crew  in  the  loop  system  assessments. 

There  are  three  overall  objectives  (or  Measures  of  Effectiveness  (MOE))  by  which  the 
project’s  progress  shall  be  measured  by  the  DND  Senior  Review  Board. 

a.  MOE#l:  Establish  a  distributed  high  fidelity  networked  CH146  virtual 
environment. 

b.  MOE#2:  Contribute  three  post  graduate  level  studies  in  SA  to  the  open 
literature. 

c.  MOE#3:  Conduct  one  validated  operational  test/e  valuation  to  demonstrate  the 
role  of  simulation  in  the  acquisition  process. 

The  objectives  of  MOE#  1  and  MOE#3  are  being  addressed  in  large  part  through  the 
TAMSS  Systems  Integrator  Contract. 

The  TAMSS  Situational  awareness  studies  contract  with  CACR  addresses  objective 
MOE#2.  Work  by  the  CACR  will  include  a  series  of  three  SA  studies  to  examine  pilot  crew 
options  in  a  virtual  simulation.  As  part  of  the  SA  studies  contract,  a  CHI  46  Griffon  Networked 
Tactical  Simulator  (NTS)  has  been  built  for  the  CACR.  The  CH146  NTS  allows  the  crews  to  be 
immersed  in  a  virtual  simulation  of  a  tactical  environment.  The  task  of  CMC  Electronics  is  to 
integrate  the  ERSTA  HFE  mock-up  with  the  CH146  NTS  to  augment  the  SA  Studies  conducted 
by  CACR. 


The  ERSTA  HFE  mock-up  was  designed  to  conduct  human  factors  experimentation 
of  the  operator  station  using  an  aircraft  FLIR  system.  For  the  purpose  of  the  CACR  studies,  a 
simulated  ERSTA  system  is  required.  This  ERSTA  system  will  have  to  be  a  generic 
representation,  since  the  system  has  not  been  purchased  for  the  CHI 46  fleet. 


Version  002 


1 


26  Apr  04 


CMC  ELECTRONICS  INC. 


DOC  NO 

HUMAN  FACTORS  ENGINEERING  409-A64025-002 

1.2  Functional  Prototype  Components 

The  ERSTA  HFE  mock-up  will  provide  the  following  functional  components: 

a.  A  sensor  simulation  system  to  simulate  a  Colour  Day  Television  (CDTV) 
System. 

b.  A  virtual  scene  displays  (CDTV)  to  simulate  real-time  video  display  of  sensor 
imagery. 

c.  A  tactical  map  display  to  show  infonnation  received  from  the  NTS. 

d.  A  cockpit  map  display  to  show  infonnation  received  from  the  NTS. 

e.  A  communication  system  (HLA)  to  communicate  with  NTS. 

f.  A  Centre  Console  Instrument  Display  to  show  instruments  and  a  second 
virtual  scene  (CDTV). 

g.  Forms  to  send  and  receive  messages. 

h.  A  system  manager  to  start  up  all  of  the  above  components. 


1.3  External  Interfaces 

Since  the  prototype  software  is  driving  graphics,  the  workload  is  relatively  heavy.  To 
divide  the  workload,  three  personal  computers  are  used  to  implement  different  tasks.  The  three 
personal  computers  are  connected  through  a  network  and  all  the  external  interfaces  are  identified 
in  the  drawing  Figure  1  as  connections  in  or  out  of  the  four  computers.  These  include  a  Joystick, 
a  Keyboard,  Video,  Mouse  (KVM)  switch  and  a  hub. 


1.4  States  and  Modes 

There  are  four  states  for  the  TAMSS  rapid  prototype:  Off,  Initialization,  Run  and 

Shutdown. 


The  operator  is  required  to  transit  from  the  Off  state  to  Initialization  State  by  starting 
the  System  Manager.  The  system  automatically  transitions  from  the  Initialization  State  to  Run 
State.  And  the  operator  is  required  to  initiate  the  state  transition  from  Run  to  Shutdown  by 
pressing  a  designated  shutdown  key. 

There  are  no  special  modes  for  the  above  four  states. 
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2  HARDWARE  ARCHITECTURE 

2.1  Component  Overview 

The  ERSTA  HFE  mock-up  will  include  the  following  hardware  components: 

a.  Three  desktop  computers 

b.  Four  flat  panel,  colour  monitors  (including  two  touch-screen  monitors) 

c.  Two  standard  keyboards  and  mice 

d.  A  KVM  switch 

e.  A  Hub 

f.  Two  Joysticks  (One  Joystick  and  One  Game  Pad) 

g.  One  Cereal  Box 

Additionally,  two  instrument  displays  will  be  added  to  the  NTS  as  shown  in  Figure  1. 
Except  for  the  inclusion  of  a  sensor  video  window  in  the  centre  display,  these  two  cockpit 
displays  are  not  part  of  the  ERSTA  mock-up  and  therefore  will  be  not  described  here. 

2.2  Desktop  Computers 

One  computer  is  needed  to  drive  CDTV  sensor  images  with  a  resolution  of 
approximately  640  x  480  (refer  to  Section  2.3)  at  a  minimum  rate  of  60  frames/second.  To 
achieve  this  rate,  large  process  memory  (RAM),  large  video  memory  and  high-speed  computers 
are  needed.  Single  Pentium3  1.8GHZ  CPU,  1024M  RAM,  128M  video  memory  computers  are 
selected  to  drive  the  visual  simulation  images. 

Another  two  computers  need  to  drive  map  displays  with  a  resolution  of  approximately 
1024  x  768  and  800  x  600  (refer  to  Section  2.3).  The  maps  will  be  shown  on  the  Tactical  Map 
display  and  Cockpit  Map  Display  separately.  To  meet  this  workload,  dual  Pentium3  1.8GHZ 
CPU,  1024M  RAM,  128M  video  memory  computers  are  selected  to  drive  the  two  map  graphics. 
Sensor  simulation,  Video  Image  and  Forms  will  be  developed  and  run  on  one  of  these  two 
computers  (Refer  to  3  Software  Architecture). 


2.3  Flat  Panel,  Colour  Touch-screens 

The  are  two  ViewSonic  VP  15 1  monitors  (Touch-Screen).  One  monitor  will  be  used 
as  Tactical  Map  Display,  set  at  1024  x  768  resolution  to  display  Tactical  Map.  The  other  one 
will  be  used  as  Visual  Simulation  Display,  set  at  768  *  1024  (Rotated)  to  display  the  sensor 
image  (CDTV)  and  forms  in  the  up-down  layout  (Touch  function  is  not  used  for  this  display)  . 

The  NTS  system  has  two  NEC  201  OX  monitors  to  be  used  as  Pilot  Flight  Instruments 
Display  and  Co-pilot  Flight  Instruments  Display.  These  two  displays  are  drawn  in  the  Figure  1 
TAMSS-ERSTA  Prototype  Architecture  and  Figure  3  TAMSS-ERSTA  System  Configuration 
since  they  will  be  used  in  conjunction  with  other  displays.  To  maintain  consistency  among  the 
instrument  displays,  a  NEC  201  OX  monitor  is  chosen  as  the  Centre  Instrument  Display. 
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Figure  1  TAMSS-ERSTA  Prototype  Architecture 
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A  12”  Touchtek  monitor  (Touch  Screen)  is  used  as  Cockpit  Map  Display  setting  at 
800  *  600  to  display  the  Cockpit  Map 


2.4  Standard  keyboards  and  mouse 

Two  standard  PC  keyboard  and  a  standard  mouse  will  be  connected  to  the  PCs  using 
the  PS/2  input.  One  keyboard  and  the  mouse  will  be  connected  to  Harry  (Figure  3)  for  the  Form 
inputs  and  starting  the  applications.  The  other  keyboard  will  be  connected  to  Hagrid  (Figure  3)  to 
shutdown  the  applications.  Using  a  separate  keyboard  for  shutdown  is  to  prevent  the  operators 
from  shutting  down  the  system  by  mistake. 


2.5  KVM  system 

A  SwitchView  SV83 1  KVM  system  is  connected  to  the  PCs.  The  purpose  of  using 
the  switch  is  to  easily  switch  between  the  computers.  It  is  intended  only  for  use  in  development 
and  testing. 

2.6  Hub 

The  hub  is  used  to  set  up  the  network  connecting  the  ERSTA  HFE  mock-up  to  the 
NTS  and  enabling  the  internal  communication  within  the  ERSTA  HFE  mock-up. 


2.7  Joysticks 

There  are  two  joysticks  used  for  the  ERSTA  HFE  mock-up.  One  is  for  the  Cabin 
operator  and  the  other  is  for  the  co-pilot.  An  existing  bilateral  handgrip  joystick  will  be  used  for 
the  Cabin  operator.  The  analog  and  discrete  outputs  of  the  joystick  are  captured  by  a  cereal  box 
(refer  to  Section  2.8)  and  forwarded  via  RS-232  serial  communications  to  the  sensor  computer. 
A  Logitech  Wingman  Rumble  Pad  will  be  used  for  the  co-pilot  and  will  connect  to  the  sensor 
computer  (Harry,  Figure  3)  using  USB.  The  original  switch  functionality  was  defined  in  the 
Human  Engineering  Design  and  Approach  Document  -  Operator  (HEDAD-O  [Ref  1]). 
However  some  functions  need  to  be  redefined  based  on  the  ERSTA  mock-up  requirements,  and 
mapped  to  the  two  different  control  devices. 


2.8  Cereal  Box 

There  is  a  cereal  box  used  to  interface  the  cabin  joystick  to  the  sensor  computer.  The 
cereal  box  transforms  and  packages  the  control  input  data  for  transmission  to  the  sensor 
computer  over  an  RS-232  serial  communication  interface. 
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3  SOFTWARE  ARCHITECTURE 

3.1  Component  Overview 

There  are  two  basic  requirements  for  the  ERSTA  mock-up: 

-  One  visual  simulation  images  will  be  shown  on  the  Visual  Simulation  Display  and 
Centre  Instrument  Display  based  on  the  sensor  simulation. 

-  Two  map  graphics  will  be  shown  on  the  Tactical  Map  display  and  Cockpit  Map 
Display. 

To  meet  the  basic  requirements,  one  visual  simulation  processes  will  be  developed  to 
simulate  a  CDTV  image  so  that  it  can  be  captured  and  displayed  in  different  sizes  and  layouts  on 
the  different  monitors.  Another  two  processes  will  be  developed  to  capture  and  show  the  image 
on  the  displays. 

The  whole  TAMSS-ERSTA  system  software  therefore  has  twelve  major  components 
including  nine  executable  processes  and  three  databases. 

Processes: 

a.  System  Manager 

b.  HLA 

c.  Sensor  Simulation 

d.  CDTV  Visual  Simulation 

e.  Tactical  Map 

f.  Cockpit  Map 

g.  Forms 

h.  Cockpit  Video  (Video  Capture) 

i.  Visual  Images  (Video  Capture) 

Databases: 

a.  Configuration  Database 

b.  Visual  Simulation  Database 

c.  Map  Database 

Experimental  data  will  be  recorded.  The  data  recording  process  is  on  the  NTS  side  so 
it  is  not  included  in  the  TAMSS-ERSTA  system  software.  However,  the  sensor 
simulation  will  send  sensor  and  visual  data  to  the  recording  process.  For  this  reason 
the  data  recording  process  is  still  drawn  in  the  software  architecture. 

Figure  2  illustrates  the  software  architecture  which  identifies  the  links  of  all  the 
components.  The  software  will  be  implemented  using  C/C++,  VEGA,  OpenGL  and  VR_LINK. 

3.1.1  C/C++ 

C/C++  is  the  fundamental  tool  for  the  software  development.  It  has  the  following 
good  points: 
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-  C/C++  allows  the  manipulation  of  bits,  bytes  and  addresses-  the  basic  elements  with 
which  the  computer  functions. 

-  Its  portability  makes  it  possible  to  adapt  software  written  for  one  type  of  computer  to 
another. 

-  All  other  application  software  VEGA,  OpenGL  and  VR  LINK  are  built  on  C/C++. 
C/C++  is  chosen  as  the  basic  tool  to  develop  the  TAMSS-ERSTA  rapid  prototype  software. 


3.1.2  VEGA 

VEGA  is  a  software  environment  for  virtual  reality  and  real-time  simulation 
applications.  By  combining  advanced  simulation  functionality  with  an  easy-to-use  tool,  VEGA 
provides  a  means  of  constructing  sophisticated  applications  quickly  and  easily.  It  has  the 
following  features: 

-  Well  used  in  the  airline  transportation,  aircraft  manufacturing,  space  and  defence 
industries 

-  Comes  with  an  extensible  point-and-click  graphical  user  interface,  enabling  changes 
to  significant  application  parameters  without  coding  or  re-compiling. 

-  Available  in  multi-process  (MP)  and  single-process  (SP)  configurations  and  offers  a 
low  cost  solution  for  systems  with  a  single  CPU  and  supports  the  development  of 
applications  using  a  single  process  runtime  model. 


Based  on  these  features  VEGA  is  the  major  development  tool  for  the  Out-The- 
Window  (OTW)  scene  of  the  CH146  Griffon  Networked  Tactical  Simulator  (NTS)  which  has 
been  built  for  the  CACR.  To  be  consistent  with  the  existing  NTS  system  and  build  high  fidelity 
sensor  simulation  images,  VEGA  is  chosen  as  a  development  tool  for  CDTV  and  TIS  visual 
simulation. 


The  Vega  FX  module  is  also  being  used  to  provide  the  simulation  of  entity 
interactions  such  as  smoke  and  flame  when  an  object  is  damaged  within  the  scenario. 


3.1.3  OpenGL 

OpenGL  is  an  environment  for  developing  portable,  interactive  2D  and  3D  graphics 
applications.  It  is  the  industry's  most  widely  used  and  supported  2D  and  3D  graphics  application 
programming  interface  (API),  bringing  thousands  of  applications  to  a  wide  variety  of  computer 
platforms.  VEGA  is  built  upon  OpenGL.  It  has  the  following  advantages: 

-  Industry  standard  with  broad  support 
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-  Stable  and  available  on  a  wide  variety  of  platforms. 

-  Reliable  and  portable 

-  Evolving 

-  Scalable 

-  Easy  to  use 

-  Well-documented 

For  these  advantages  and  features,  OpenGL  is  selected  to  implement  Tactical  and 
Cockpit  maps  and  all  the  overlays  on  the  CDTV  and  TIS  displays. 
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3.1.4  VR  LINK 

The  VR  LINK  library  is  used  to  implement  HLA  interface  for  communications  with 
the  NTS  specified  in  the  SOW  [Ref  3]).  HLA  is  the  industry  standard  method  of  inter-simulation 
communications.  Additionally,  the  use  of  HLA  will  facilitate  the  future  use  of  a  modified 
ERSTA  prototype  with  other  simulations. 


3.2  System  Manager 

The  task  of  the  system  manager  is  to  start  the  other  processes  locally  or  remotely. 
“Locally”  means  to  start  processes  on  the  same  computer.  “Remotely”  means  to  start  the 
processes  on  different  computers.  It  saves  time  and  trouble  that  will  be  required  to  start  each 
process  separately.  The  process  is  implemented  using  C/C++. 


3.3  HLA 

The  HLA  interface  is  intended  to  handle  all  HLA  communications.  The  data  received 
via  HLA  will  then  be  placed  in  an  internal  format  that  is  appropriate  for  use  by  other  modules. 
This  module  will  be  written  in  C/C++  using  the  appropriate  VR_LINK  libraries. 

3.4  Sensor  Simulation 

The  sensor  simulation  process  simulates  the  functions  of  a  real  sensor.  The  process 
has  two  modules:  I/O  handler  and  sensor. 


3.4.1  I/O  Handler 

I/O  handler  takes  the  analog  and  discrete  outputs  from  the  cereal  box  which  captures 
joystick  and  switch  events,  then  transfers  the  data  to  the  Sensor  Simulation  Module.  This  module 
is  implemented  in  C/C++. 
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3.4.2  Sensor 

The  Sensor  emulates  the  functionality  of  the  sensor  ball.  The  functions  include: 

-  Pan  and  tilt 

Use  joystick  to  control  pan  and  tilt. 

-  Zoom 

Be  able  to  zoom  in  and  out. 

-  Rangefinding 

Determine  the  range  of  anything  in  the  boresight. 

-  Slew  to  Aircraft  Reference  Position  (ARP). 

The  sensor  is  able  to  slew  to  a  specified  position  with  respect  to  the  nose  of  the 
aircraft. 

-  Slew  to  Position  on  the  Map. 

The  sensor  is  able  to  slew  to  a  specified  position  on  the  map. 

-  Auto-tracking 

The  sensor  is  able  to  track  still  and  moving  objects  without  pan/tilt  control. 

-  Designating 

Find  the  target  position  to  be  designated. 

The  sensor  module  is  implemented  using  C/C++. 
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3.5  CDTV  Visual  Simulation 

The  CDTV  Visual  Simulation  process  simulates  the  image  from  a  Color  Day 
TeleVison  which  is  installed  inside  the  sensor  ball.  It  has  the  following  functions: 

-  Functions 

a.  Dynamically  add,  locate  and  remove  entities  of  ground  vehicles,  aircraft  and 
missiles. 

b.  Make  smoke  and  fire  when  the  entities  are  damaged. 

c.  Zoom  in  and  out 

d.  Find  an  object  when  the  auto-tracker  run  mode  is  selected 

e.  Provide  corresponding  visual  of  the  sensor. 

-  Overlays 

a.  Sensor  position  and  orientation 

b.  Reticule 

c.  Track  Window 

d.  Zoom  box 

e.  Run  mode 

f.  Sensor  controller 

g.  Range  and  Bearing 

h.  Zoom 

i.  Designating 

The  CDTV  Visual  Simulation  is  implemented  in  C/C++  using  VEGA  and  OpenGL 

libraries. 


3.6  Visual  Image 

The  process  grabs  video  signals  produced  by  the  CDTV  Visual  Simulation  and 
displays  the  video  image  on  the  Visual  Simulation  Display.  The  functions  of  this  process 
includes: 


-  Display  Images 

Display  the  CDTV  image. 

The  process  will  use  the  ATI  Video  Card  and  Microsoft  DirectX9.0. 


3.7  Cockpit  Video 

This  process  grabs  video  signals  produced  by  the  CDTV  Visual  Simulation  and 
displays  the  video  image  on  the  Cockpit  Instrument  Display.  The  functions  of  this  process 
includes: 
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-  Display  Images 

Display  the  CDTV  image. 

The  process  will  use  the  ATI  Video  Card  and  Microsoft  DirectX9.0  and  is  started 
with  the  Centre  Instrument  Display  which  is  on  the  NTS  side  of  the  Network. 

3.8  Tactical  Map 

The  map  process  provides  a  two-dimensional  map  with  overview  of  the  tactical  and 
operational  areas  of  interest.  The  map  is  based  on  military  map  formats.  The  tactical  map 
implements  the  following  functions: 

Scale  and  Magnification 

The  user  is  able  to  zoom  in  to  look  at  a  small  area  or  zoom  out  to  look  at  the 
surroundings. 

Tactical  Features 

Display  and  identify  phase  lines,  points,  routes,  tracks,  borders  and  zones. 

North  Up  and  Heading  Up  Orientation  Mode 

North  up  mode  always  keeps  north  in  the  up  direction.  The  aircraft  symbol  rotates 
when  the  heading  changes.  Heading  up  mode  always  keeps  the  aircraft  heading  in  the 
up  direction.  The  map  rotates  when  the  heading  changes. 

Aircraft  Symbol 

A  symbol  is  used  to  clearly  identify  the  aircraft  position,  heading,  sensor 
orientation  and  field  of  regard. 

Map  Overlays 

Import  and  export  tactical  traces  including: 

a.  enemy  or  threat  disposition; 

b.  friendly  disposition; 

c.  control  measures  (phase  lines,  unit  or  formation  boundaries  etc). 

d.  obstacle  Plan; 

e.  tactical  Update  and  /or  Handover;  and 

f.  custom  “personal”  overlay  created  to  allow  the  user  to  de-clutter. 

The  map  process  is  implemented  in  C/C++  using  the  OpenGL  library. 


3.9  Cockpit  Map 

Like  the  Tactical  Map,  the  Cockpit  Map  process  provides  a  two-dimensional  map 
with  an  overview  of  the  tactical  and  operational  areas  of  interest.  It  has  the  same  functionality  as 
the  Tactical  Map  (Refer  to  Section  3.8).  However  the  Cockpit  Map  can  be  manipulated 
independently  from  the  Tactical  Map. 

The  map  process  is  implemented  in  C/C++  using  the  OpenGL  library. 
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3.10  Forms 

The  process  provides  a  graphic  user  interface  to  fill  out  a  report  at  run  time.  The 
process  will  be  implemented  using  Visual  Basic.  The  forms  include: 

-  Main  form  to  provide  menu  for  choosing  other  forms, 

-  Contact  form, 

-  Close  Air  Support  (CAS)  form. 


3.11  Database 

All  data  manipulated  by  the  above  components  are  accessed  or  saved  in  a  database. 
The  database  includes  three  major  parts:  Configuration  Files,  Visual  Simulation  Database,  Map 
Database 


3.11.1  Configuration  Files 

The  configuration  files  store  the  initial  parameters  used  by  the  above  processes. 

3.11.2  Visual  Simulation  Database 

The  database  stores  the  scene  database  (Gagetown  terrain)  and  all  the  entity  models 
that  will  be  used  in  the  Visual  Simulation  module. 

3.11.3  Map  Database 

The  database  stores  the  Gagetown  map,  all  symbology  and  the  geographic  data  for  the 
tactical  features  used  in  the  Tactical  Map  and  Cockpit  Map  process. 
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4  SYSTEM  CONFIGURATIONS 

As  shown  in  the  hardware  architecture  (Refer  to  Section  2  Hardware  Architecture), 
three  computers  will  be  used  to  support  the  whole  system.  The  computers  will  communicate  with 
each  other  over  Ethernet.  The  joysticks  are  connected  into  the  sensor  computer  through  an 
RS232  serial  port  and  one  USB  port.  All  the  processes  except  Cockpit  Video  will  be  started  by 
System  Manager.  The  processes  are  designed  to  be  easily  moved  to  the  other  computers,  that  is, 
to  remove  a  process  executable  file  from  one  computer  and  install  it  on  another  computer.  The 
system  configuration  is  shown  as  Figure  3. 
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Figure  3  Stage  2  (Experiment  #3)  TAMSS-ERSTA  System  Configuration 
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Figure  4  is  the  physical  equipment  layout  for  the  experiments. 
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5  DIRECTORY  STRUCTURE 

Windows  Version  Control  System  (WinCVS),  version  1.3,  will  be  used  to  control 
access  to  source  code  and  to  manage  internal  releases.  Figure  5  shows  the  directory  structure. 

All  the  computers  will  use  the  same  configuration. 


Common\:  Source,  header  and  library  files  used  for  all 

processes 

Database^ 

Map\: 

Geodata\:  All  geographic  data 
Maps\:  All  maps 
Symbology\:  All  symbologies 
Visuah: 

EntityV  All  entities 
Scene\:  Gagetown  Scene  database 
Documentation^ 

ArchSpec\:  Architecture  Specification 
DevPlan\:  Development  Plan 
VDD\:  Version  Description  Documentation 
Included  Include  files  from  all  the  process 
Library\: 

Process\: 

CDTV\:  CDTV  Workspace 
CDTV_video\:  CDTV  video  Workspace 
CockpitMap\:  CockpitMap  Workspace 
Forms\:  Fonns  Workspace 
HLA\:  HLA  Workspace 
Sensor\:  Sensor  Workspace 
SysMag\: 

Launched:  Launcher  Workspace 
Manager\:  Manager  Workspace 
TacMap\:  Tactical  Map  Workspace 


Figure  5  Directory  Structure  for  TAMSS-ERSTA  SA 
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6  SOFTWARE  DEVELOPMENT  ENVIRONMENT 

6.1  Hardware 

Three  Pentium  III  Computers  with: 

single  or  dual  processor, 

64  MB  RAM  (minimum), 

OpenGL-based  graphics  card  with  at  least  32  MB  texture  memory. 

The  Sensor/TacMap/VideoImage/Form  computer  must  have  two  serial  ports,  one 
USB  port  and  one  ATI  video  card. 

The  CDTV  computer  must  have  video  output. 

The  CockpitMap/CockpitVideo  must  have  one  serial  port,  one  ATI  video  card. 

One  Video  splitter 

Two  Standard  Keyboards 

One  Track  ball  or  mouse 

Two  Joysticks 

Two  touch-screen  monitors  and  two  standard  monitors. 

One  Hub 


6.2  Operating  System 

Windows  2000 

6.3  Software 

VEGA  Developer  licence  and  VEGA  running  licence  with  Special  Effect  (FX) 
module 

-  MAK  VRLINK  Licence 

-  DMSO  RTI 1 .3NG  Version  6 
Microsoft  Visual  C++  V6.0  or  later 
OpenGL  Libraries 

Microsoft  Visual  Basic  V6.0  or  later 
Touch  display  driver 
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Display  rotation  software 

6.4  Data  and  Databases 

Gagetown  Scene  Database 

Entity  Models  in  Open  Flight  Fonnat 

Gagetown  Maps 

Military  Standard  Symbologies 

Geographic  Data  for  Tactical  Features 
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1  CONFIGURATION 

1.1  Overview 

The  TAMSSSA  rapid  prototype  project  is  being  conducted  to  support  the  Situational 
Awareness  (SA)  studies  for  the  Tactical  Aviation  Mission  System  Simulation  (TAMSS)  SA 
Technology  Demonstration  Project  (TDP).  The  studies  comprise  three  experiments: 

•  Experiment  #1  uses  only  a  Griffon  cockpit  simulation. 

•  Experiment  #2  requires  the  addition  of  the  cabin  mission  system,  called  the 
Electro-Optic  Reconnaissance,  Surveillance  and  Target  Acquisition  (ERSTA) 
system.  In  the  first  stage  of  ERSTA  development,  the  focus  was  on  the 
hardware  setup  and  software  foundation.  The  software  components  included 
in  the  ERSTA  system  for  Experiment  #2  are: 

Sensor  Simulation  System 

Colour  Day  Television  (CDTV) 

Tactical  Map 

Cockpit  Map 

High  Level  Architecture  (HLA) 

Video  Capture  modules 
Contact  Form 
System  Manager 

•  Experiment  #3  requires  more  interaction  (firing)  and  allows  the  Mission 
Commander  to  operate  the  sensor  in  the  ERSTA  system.  The  major  functions 
added  into  the  ERSTA  system  for  Experiment  #3  are: 

Artillery  firing  and  entity  status 

Additional  Joystick  (Game  Pad)  for  the  Mission  Commander  to  control  the 
ERSTA  system 

Map  grid  for  the  operators  easy  to  get  positions 

Close  Air  Support  (CAS)  Fonn,  Fire  Mission  Form  and  Main  Fonn 

The  detailed  functions  for  Experiment  #2  and  additional  functions  for  Experiment  #3 
in  each  process  are  listed  in  Table  1-1. 

Although  the  Thermal  Imager  (TI)  simulation  described  in  the  system  architecture 
[Ref  1]  was  planned  to  be  added  to  the  ERSTA  system  for  Experiment  #3,  this  requirement  was 
dropped  by  the  customer. 
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Table  1-1  lists  the  completed  software  processes  and  associated  functions. 

Table  1-1  Completed  Software  Processes  and  Associated  Functions 


Process 

Functionality 

Experiment  #2 

Experiment  #3 

System 

Manager 

Start  up  all  other  processes 

Start  Sensor,  CDTV, 
HLA,  Tactical  Map, 
Cockpit  Map,  Video 
Image,  Forms 

HLA 

Interface 

Communicate  with 
Networked  Tactical 
Simulator  (NTS) 

Communicate  with 

NTS  for: 

.  Ownship  position 
and  orientation 

.  Entities  interaction 
for  creating  and 
removing 

.  Entities  position 
and  orientation 

Entity  interaction  for 
the  entity  status 

Sensor 

Simulation 

Simulate  the  functions  of  a 
real  sensor 

Control: 

.  One  input/output 
(I/O)  handler 

.  Pan  and  Tilt 

.  Zoom  of  CDTV 

.  Range  finding 

.  Slew  to  Aircraft 
Reference  Position 
(ARP) 

.  Slew  to  a  position 
on  the  map 

.  Auto-tracking 

.  Designating 

.  Target  marking 

Control: 

.  A  second  I/O 
handler 

.  I/O  controllers 
interaction 

CDTV  Visual 
Simulation 

Simulate  the  image  from  a 
CDTVg| 

.  Get  and  move  scene 

.  Create  and  remove 
entities 

.  Calculate  range 

.  Find  object  for 
auto-tracking 

.  Visual 

representation  for 
artillery  firing, 
bomb  detonating 
and  burning  entities. 

.  Visual 

representation  for 
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Table  1-1  Completed  Software  Processes  and  Associated  Functions 


Process 

Functionality 

Experiment  #2 

Experiment  #3 

.  Overlays 

the  crashed  entities. 

•  Field  of  regard 

Tactical  Map 
Cockpit  Map 

A  two-dimensional  map 
with  an  overview  of  the 
tactical  and  operational 
areas  of  interest 

.  Moving  Map 

.  Scale  and 
Magnification 

.  Tactical  Features 

.  North  up  orientation 
.  Aircraft  Symbol 
.  Overlays 

•  Heading  up 
orientation 

•  Orientation  model 
switching 

•  Map  grid  in 

Universal 

Transverse  Mercator 
(UTM)  coordinate. 

•  Map  grid  ON-OFF 
switch 

•  Map  flip  (only 
Cockpit  Map) 

Forms 

Graphical  User  Interface 
(GUI)  to  receive  and  send 
messages 

.  Contact  Form 

•  Main  Form 

•  CAS  Form 

•  Fire  Mission  Form 

Visual  Image 
Cockpit 

Video 

Grab  video  signals  and 
display  the  video  image  on 
the  Visual  Simulation 
Display  and  the  Cockpit 
Console  Instrument 

Display 

.  Grab  CD  TV  image 

Data 

Recording 

Record  data 

.  Record  Data  for 
Experiment  #2 

•  Record  Data 
Experiment  #3 
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Figure  1-1  shows  the  system  configuration  for  Experiment  #3. 


Figure  1-1  Stage  2  (Experiment  #3)  TAMSS-ERSTA  System  Configuration 


1.2  Software  Components 

Table  1-2  lists  the  software  components  that  must  be  used  in  order  to  start  up  and  run 
the  TAMSS  Rapid  Prototype  application.  All  the  executable  files,  configuration  files  and  data 
files  are  stored  in  the  following  location  of  the  corresponding  computer:  C:\TAMSS\ 
Experiment3\Project\TAMSS_SA\.  All  the  source  code  is  saved  on  the  company  MKS.  The 
data  recording  process  saves  data  in  a  database  on  the  NTS  side  of  TAMSS. 
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Table  1-2  TAMSS  Software  Components 


Software 

Component 

Location 

Functionality 

Comments 

Manager.exe 

Harry-> 

C:\TAMSS\Experiment3 

\Project\TAMSS_SA\Process\ 

Manager) 

Start  and  run  all 
the  ERSTA 
applications 

A  shortcut,  named 
Manager,  is  on  the 
desktop  on  Harry. 

See  footnote  '. 

Scenario  1. xml 
Scenario2.xml 
Scenario3.xml 
Scenario4.xml 

Harry-> 

C:\TAMSS\Experiment3 

\Project\TAMSS_SA\Process\ 

Manager) 

Set  up  the  path  of 
all  the  software 
processes  and 
scenario  numbers 

These  files  must  be 
in  Extensible 

Markup  Language 
(XML)  format  and 
must  be  in  the  same 
directory  as 
Manager.exe 

Launcher.exe 

Harry,  Hagrid,  Hennione-> 
C:)TAMSS)Experiment3 
)Project)TAMSS_SA)Process) 
Launcher) 

Listen  to  the 
message  from 
Manager.exe  to 
start  processes  on 
local  computers 

A  shortcut,  named 
Launcher,  is  in  the 
Start  menu.  The 
process  will  run 
automatically  when 
the  user  logs  into 
the  computer 

Sensor.exe 

Harry-> 

C:)TAMSS)Experiment3 

)Project)TAMSS_SA)Process) 

Sensor) 

Simulate  a  real 

sensor 

Map.exe 
(Tactical  Map) 

Harry-> 

C:)TAMSS)Experiment3 
)Project)TAMSS_SA)Process) 
Tactical  Map) 

Tactical  map  with 
overview  of  the 
tactical  and 
operational  areas 
of  interest  and 
overlays 

Maps  Folder 
(12  Files  in  bitmap 
[.bmp]  format) 

Harry-> 

C:)TAMSS)Experiment3 
)Proj  ect)T  AM  S  S_S  A) 
Database)Map) 

Gagetown  map  in 
tiles 

The  map  files  must 
have  a  .bmp 
extension 

Symbology  Folder 
(53  Files  in  tga 
format) 

Harry-> 

C:)TAMSS)Experiment3 
)Proj  ect)T  AM  S  S_S  A) 
Database)Map) 

Symbology  (image 
files)  for  map 
overlays 

The  symbology 
files  must  have  a 
.tga  extension 

1  Manager.exe  needs  a  scenario  number  as  an  argument.  The  number  is  currently  set  up  in  the  shortcut  icons  of 
Manager  (SI,  S2,  S3,  S4  on  the  desktop).  Refer  to  Appendix  A  “Start-up  Procedure  for  the  ERSTA  Mock-up”  for 
instructions  on  how  to  run  the  four  scenarios. 
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Table  1-2  TAMSS  Software  Components 


Software 

Component 

Location 

Functionality 

Comments 

Geodata  Folder 
aco  N.xml 
custom  N.xml 
tactical  N.xml 
(N:  1-4) 

Harry-> 

C:\TAMSS\Experiment3 
\Project\TAMSS  SAYDatabase 
YMap\ 

Geographic  and 
Symbology  data 
for  map  overlays 

The  geodata  files 
must  have  a  .xml 
extension 

CDTV_Video.exe 

Harry-> 

C:\TAMSS\Experiment3 
YProjectYTAMSS  SA\Process\ 
CDTV_TIS_Video\ 

Grab  CDTV  video 
signal  and  display 
it  on  Visual 
Simulation 

Display 

ERS  T  AF  orms .  exe 

Harry-> 

C:\TAMSSYExperiment3 

YProject\TAMSS_SA\Process\ 

Forms) 

ERST  A  Forms 

In  Visual  Basic 

CDTV.exe 

Hagrid-> 

C:\TAMSS\Experiment3 
\Project\TAMSS  SA\Process\ 
CDTV 

Simulate  the  image 
from  a  CDTV 

2 

See  footnote  ~ 

Gagetownl.adf 

Gagetown2.adf 

Gagetown3.adf 

Gagetown4.adf 

Hagrid-> 

C:\TAMSS\Experiment3 
\Proj  ect\T  AM  S  S_S  A\ 
Process\Launcher\ 

Application  Define 
File  to  define 
system  and 
environment 
variables  for  each 
scenario 

The  files  must  be  in 
the  same  folder  as 
Launcher.exe 

Gagetown  Folder 
(20  files  in  fst 
format) 

Hagrid-> 

C:\TAMSS\Experiment3 
\Project\TAMSS  SA\Database 
\Visual\Scene\ 

Gagetown 
database  in  tiles 

2  CDTV.exe  needs  a  scenario  number  as  an  argument.  The  number  is  set  up  in  the  scenarioN.xml  (N:  1-4)  files. 
The  user  does  not  need  to  change  the  files  for  Experiment  #3. 
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Table  1-2  TAMSS  Software  Components 


Software 

Component 

Location 

Functionality 

Comments 

GoodEntity  Folder 
(20  model  folders. 
Each  folder 
includes  fdes  in 
.fit,  .rgb,  and 
.rgb.attr  format) 

Hagrid-> 

C:\TAMSS\Experiment3 
\Project\TAMSS  SAYDatabase 
\Visual\Entity\ 

Entity  models 

.  Files  with  a  .fit 
extension  are  the 
model  files. 

.  Files  with  a  .rgb 
extension  are  the 
texture  files  for 
the  models. 

.  Files  with  a 
.rgb.attr 

extension  are  the 
texture  attribute 
files  for  the 
models. 

HLATest.exe 

Hagrid-> 

C:\TAMSSYExperiment3 
YProjectYTAMSS  SA\Process\ 
HLA 

Communicate  with 
NTS 

VegaStage.ini 

Hagrid-> 

C:\TAMSSYExperiment3 

YProject\TAMSS_SA\Process\ 

SysMag\Launcher\ 

The  configuration 
file  for 

HLATest.exe 

The  file  must  be  in 
the  same  folder  as 
Launcher.exe 

VR  Link,  fed 

Hagrid-> 

C:\TAMSSYExperiment3 

YProject\TAMSS_SA\Process 

\SysMag\Launcher\ 

The  Federation 
definition  file  for 
HLATest.exe 

The  file  must  be  in 
the  same  folder  as 
Launcher.exe 

RTIStage.rid 

Hagrid-> 

C:\TAMSS\Experiment3 

YProject\TAMSS_SA\Process\ 

SysMag\Launcher\ 

Run  Time 
Infrastructure 

(RTI) 

initialization  Data 
file  to  define  data 
required  by  the 

RTI 

The  file  must  be  in 
the  same  folder  as 
Launcher.exe 

Disegtype 

Hagrid-> 

C:\TAMSS\Experiment3 

YProject\TAMSS_SA\Process\ 

SysMag\Launcher\ 

The  entity  model 
map  file  for 
HLATest.exe 

The  file  must  be  in 
the  same  folder  as 
Launcher.exe 

Runlm.exe 

Hagrid-> 

C:\bin\ 

Licence  manager 
file  to  run 
HLATest.exe 

This  file  must  be 
running  before 
starting 

HLATest.exe. 
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Table  1-2  TAMSS  Software  Components 


Software 

Component 

Location 

Functionality 

Comments 

Lmdown.exe 

Hagrid-> 

C:\bin\ 

Shutdown  the 
licence  manager 
file 

Run  this  executable 
file  only  when 
required  to  force  a 
licence  manager 
shutdown 

RunCDTVSl.bat 

RunCDTVS2.bat 

RunCDTVS3.bat 

RunCDTVS4.bat 

Hagrid-> 

C:\TAMSS\Experiment3 

\Project\TAMSS_SA\Process\ 

SysMag\Launcher\ 

A  batch  file  to  set 
path  and  start 

CDTV  and  HLA 
for  each  scenario 

The  file  must  be  in 
the  same  folder  as 
Launcher.exe 

Map.exe 
(Cockpit  Map) 

Hermione-> 

C:\TAMSS\Experiment3 
\Project\TAMSS_SA\Process\ 
Cockpit  Map\ 

Cockpit  map  with 
overview  of  the 
tactical  and 
operational  areas 
of  interest  and 
overlays 

Maps  Folder 
(12  Files  in  bmp 
format) 

Hermione  -> 
C:\TAMSS\Experiment3 
\Proj  ect\T  AM  S  S_S  A\ 
Database\Map\ 

Gagetown  map  in 
tiles 

The  map  files  must 
have  a  .bmp 
extension 

Symbology  Folder 
(53  Files  in  tga 
format) 

Hermione  -> 
C:\TAMSS\Experiment3 
\Proj  ect\T  AM  S  S_S  A\ 
Database\Map\ 

Symbology  for 
map  overlays 

The  symbology 
files  must  have  a 
.tga  extension 

Geodata  Folder 
aco  N.xml 
custom  N.xml 
tactical  N.xml 
(N:  1-4) 

Hermione  -> 
C:\TAMSS\Experiment3 
\Project\TAMSS  SA\Database 
\Map\ 

Geographic  and 
Symbology  data 
for  map  overlays 

The  geodata  files 
must  have  a  .xml 
extension 

1.2.1  Building  Manager.exe 

Table  1-3  lists  the  files  included  in  the  “Manager”  project  for  Experiment  #3. 


Table  1-3  Manager.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FIFES  (.CPP) 

MKS  Server  (srveng  10:7001) 
d:\Groups\MKS\Development\HFl 

EYTAMSS  SA  TI02\process\SysMag\Manager\ 

Main.cpp 

2.1 

Send  start  up  message  to  Launcher.exe  on 
all  the  computers. 
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Table  1-3  Manager.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

HEADER  FILES(.h) 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\Include\ 

Ersta  communication 
types. h 

2.1 

Structure  and  function  definition  of  the 

UDP  communications. 

udp  comm.h 

2.1 

UDP  Communication. 

PerfTimer.h 

2.1 

Set  timer. 

LIBRARY  FILES 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\Library\ 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

udp. lib 

2.1 

Library  file  for  UDP  communication. 

libexpat.lib 

A  library  for  parsing  XML 

Before  making  any  changes  to  files  in  the  Manager  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client  computer. 

Following  a  change  to  a  file  contained  within  the  “Manager”  project,  a  new 
Manager.exe  can  be  generated  as  follows: 

Step  1.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (Manger.dsw) 

Step  2.  In  the  Build  Menu,  select  “Build  sysmag.exe”  or  press  F7. 

Step  3.  The  path  of  the  executable  file  (Manager.exe)  has  been  already  set  to 

C:\TAMSS\Experiment3\Project\TAMSS_SA\Process\Manager\.  The 
new  executable  file  should  be  generated  in  that  folder. 


1.2.2  Building  Launcher.exe 

Table  1-4  lists  the  files  included  in  the  “Launcher”  project  for  Experiment  #3. 

Table  1-4  Launcher.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.CPP) 

MKS  Server  (srveng  10:7001) 
d:\Groups\MKS\Development\HFl 

EYTAMSS  SA  TI02\process\SysMag\Launcher\ 

Main.cpp 

2.1 

Listens  to  the  messages  from  Manager.exe 
to  start  the  processes  on  its  machine. 

HEADER  FILES(.h) 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\Include\ 

Ersta  communication 
types. h 

2.1 

Structure  and  function  definition  of  the 

UDP  communications. 

udp  comm.h 

2.1 

UDP  Communication. 

PerfTimer.h 

2.1 

Sets  timer. 
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Table  1-4  Launcher.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

LIBRARY  FILES 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\Library\ 

udp.lib 

2.1 

Library  file  for  UDP  communication. 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

Before  making  changes  to  any  files  in  the  Launcher  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client  computer. 

Following  a  change  to  a  file  contained  within  the  “Launcher”  project,  a  new 
Lancher.exe  can  be  generated  as  follows: 

Step  1.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (Launcher. dsw) 

Step  2.  In  the  Build  Menu,  select  “Build  Launcher.exe”  or  press  F7. 

Step  3.  The  path  of  the  executable  file  (Lanuncher.exe)  has  been  already  set  to 

C:\TAMSS\Experiment3\Project\TAMSS_SA\Process\Launcher\.  The 
new  executable  file  should  be  generated  in  that  folder. 
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1.2.3  Building  Sensor.exe 

Table  1-5  lists  the  files  included  in  the  “Sensor”  project  for  Experiment  #3. 

Table  1-5  Sensor.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.CPP) 

MKS  Server  (srveng  10:7001) 
d:\Groups\MKS\Development\HFl 

EYTAMSS  SA  TI02\process\Sensor\ 

Main.cpp 

2.1 

Initialize  and  update  I/O  handler  and  sensor 
module. 

Joy  sick  JS.cpp 

2.1 

Gets  user  control  data  (Joystick)  from  cereal 
box. 

JoysickDX.cpp 

2.1 

Gets  user  control  data  (Game  Pad)  from 
cereal  box. 

Sensor.cpp 

2.1 

Sensor  simulation. 

HEADER  FILES(.h) 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\Include\ 

Ersta  communication 
types. h 

2.1 

Structure  and  function  definition  of  the  UDP 
communications. 

Sensor.h 

2.1 

Header  file  for  sensor.cpp. 

sensor  global. h 

2.1 

Header  file  define  global  variables  used  in 
both  Joystick.cpp  and  Sensor.cpp. 

LatLong- 

Utmconversion.h 

2.1 

Header  file  used  by  the  functions  to  convert 
between  latitude  and  longitude  and  UTM 
coordinate. 

dataCollection.h 

2.1 

Header  file  for  the  data  collection 

units. h 

2.1 

Units  conversion 

SensorDataParams.h 

2.1 

Header  file  for  data  collection 

udp  comm.h 

2.1 

UDP  Communication. 

PerfT finer. h 

2.1 

Sets  timer. 

LIBRARY  FILES 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\Library\ 

udp. lib 

2.1 

Library  file  for  UDP  communication. 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

LLvsUTM.lib 

2.1 

Library  file  to  convert  between  latitude  and 
longitude  and  UTM  coordinates. 

Before  making  changes  to  any  files  in  the  Sensor  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client  computer. 
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Following  a  change  to  a  file  contained  within  the  “Sensor”  project,  a  new  Sensor.exe 
can  be  generated  as  follows: 

Step  1.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (Sensor.dsw) 

Step  2.  In  the  Build  Menu,  select  “Build  Sensor.exe”  or  press  F7. 

Step  3.  The  path  of  the  executable  file  (Sensor.exe)  has  been  already  set  to 

C:\TAMSS\Experiment3\Project\TAMSS_SA\Process\Sensor\.  The  new 
executable  file  should  be  generated  in  that  folder. 

1.2.4  Building  Map.exe  (Tactical  map  and  Cockpit  map) 

Table  1-6  lists  the  files  included  in  the  “TacMap”  or  “CockpitMap”  project  for 
Experiment  #3. 


Table  1-6  Map.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.Cl 
MKS  Server  (srvengl 
d:\Groups\MKS\Devi 
d:\Groups\MKS\Devi 

3P) 

0:7001) 

;lopment\HFE\TAMSS  SA  TI02\process\Tactical  Map\  (for  Tactical  Map) 
;lopment\HFE\TAMSS  SA  TI02\process\Cockpit  Map\  (for  Cockpit  Map) 

area.cpp 

2.1 

Create  area  overlays 

button,  cpp 

2.1 

Create  buttons 

flightplan.cpp 

2.1 

Create  flight  plan  overlays 

GLF.cpp 

2.1 

Map  indow  frame 

glfont.cpp 

2.1 

Print  texture  texts 

label. cpp 

2.1 

Print  texture  labels 

lineseg.cpp 

2.1 

Create  line  segment  overlays 

map.  cpp 

2.1 

Get  user  control  data  from  cereal  box 

map  util. cpp 

2.1 

UTM  and  Screen  coordinate  conversions 

mytga.cpp 

2.1 

Load  textures 

overlay,  cpp 

2.1 

Draw  overlays 

overlayobject.cpp 

2.1 

Create  overlay  objects 

ownship.cpp 

2.1 

Create  ownship  overlay 

symbol. cpp 

2.1 

Create  symbol  overlays 

HEADER  FILES(.h) 
MKS  Server  (srvengl 

0:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  1 

^d 

HH 

<N 

O 

HH 

L _ _ 

area.h 

2.1 

Header  file  for  creating  area  overlays 

button.h 

2.1 

Header  file  for  creating  buttons 

ersta  communicatio 
n  types. h 

2.1 

Structure  and  function  definition  of  the  UDP 
communications 

flightplan.h 

2.1 

Header  file  for  creating  flight  plan  overlays 

GLF.h 

2.1 

Header  file  for  map  window  frame 

glfont.h 

2.1 

Header  file  for  printing  texture  texts 

label.h 

2.1 

Header  file  for  printing  texture  labels 
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Table  1-6  Map.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

LatLong- 

Utmconversion.h 

2.1 

Header  file  used  by  the  functions  to  convert 
between  latitude  and  longitude  and  UTM 
coordinate. 

lineseg.h 

2.1 

Header  file  for  creating  line  segment  overlays 

map.h 

2.1 

Header  file  for  the  map  drawing 

map  util.h 

2.1 

Header  file  for  UTM  and  Screen  coordinate 
conversions 

overlay.h 

2.1 

Header  file  for  drawing  overlays 

overlayobject.h 

2.1 

Header  file  for  drawing  overlay  objects 

ownship.h 

2.1 

Header  file  for  drawing  ownship 

mytga.h 

2.1 

Header  file  for  loading  textures 

symbol.h 

2.1 

Header  file  for  drawing  symbol  overlays 

udp  comm.h 

2.1 

Header  fiel  for  UDP  Communication 

LIBRARY  FILES 
MKS  Server  (srvengl 

0:7001)  d:\Groups\MKS\Development\HLE\TAMSS  SA  1 

Eh 

a 

(N 

O 

M 

L _ _ 

udp. lib 

2.1 

Library  file  for  UDP  communication. 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

LLvsUTM.lib 

2.1 

Library  file  to  convert  between  latitude  and 
longitude  and  UTM  coordinate. 

opengl32.1ib 

OpenGL  Library 

glu32.1ib 

OpenGL  Library 

glaux.lib 

OpenGL  Library 

ws2  32. lib 

Win  sock  Library 

libexpat.lib 

A  library  for  parsing  XML 

Before  making  changes  to  any  files  in  the  TacMap  or  CockpitMap  folder,  check  out 
the  whole  workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client 
computer. 


Following  a  change  to  a  file  contained  within  the  “TacMap”  or  “CockpitMap” 
project,  a  new  Map.exe  can  be  generated  as  follows: 

Step  1.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (Map.dsw) 

Step  2.  In  the  Build  Menu,  select  “Build  Map.exe”  or  press  F7. 

Step  3.  The  path  of  the  executable  file  (map.exe)  has  been  already  set  to 

C:\TAMSS\Experiment3\Project\TAMSS_SA\Process\<TacMap  or 
CockpitMap>\.  The  new  executable  file  should  be  generated  in  that 
folder. 
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1.2.5  Building  CDTV  Video.exe 

Table  1-7  lists  the  files  included  in  the  “CDTVJVideo”  project  for  Experiment  #3. 


Table  1-7  CDTVJVideo  .exe  Source  Code  Files 

File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.CPP) 

MKS  Server  (srveng  10:7001) 
d:\Groups\MKS\Development\HFl 

EYTAMSS  SA  TI02\process\CDTV  Video) 

Main.cpp 

2.1 

Grabs  video  signal  from  CDTV  video 
image. 

HEADER  FILES(.h) 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  1 

Id 

"O 

c 

HH 

(N 

O 

HH 

C _ _ 

Ersta  communication 
types. h 

2.1 

Structure  and  function  definition  of  the 

UDP  communications. 

udp  comm.h 

2.1 

UDP  Communication. 

PerfT finer. h 

2.1 

Sets  timer. 

LIBRARY  FILES 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  1 

t-H 

c3 

Ih 

43 

(N 

O 

HH 

C _ 1 

udp. lib 

2.1 

Library  file  for  UDP  communication. 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

Before  making  changes  to  any  files  in  the  CDTVJVideo  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client  computer. 

Following  a  change  to  a  file  contained  within  the  “CDTVvideo”  project,  a  new 
CDTV_Video.exe  can  be  generated  as  follows: 

Step  1.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (CDTVJVideo. dsw). 

Step  2.  In  the  Build  Menu,  select  “Build  CDTV_Video.exe”  or  press  F7. 

Step  3.  The  path  of  the  executable  file  (CDTV_Video.exe)  has  been  already  set  to 

C:\TAMSS\Experiment3\Project\TAMSS_SA\Process\CDTV_Video\. 
The  new  executable  file  should  be  generated  in  that  folder. 
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1.2.6  Building  ContactReport.exe 

Table  1-8  lists  the  files  included  in  the  “Forms”  project  for  Experiment  #3. 


Table  1-8  ERSTAForms  .exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.CPP) 

MKS  Server  (srveng  10:7001) 

d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\process\Forms\ 

erstaSim  prj.vbp 

2.1 

ERSTA  forms  project 

main  frm.frm 

2.1 

Main  form 

CAS  frm.frm 

2.1 

CAS  form 

fireMsn  frm.frm 

2.1 

Fire  Mission  form 

sentbox  frm.frm 

2.1 

Message  Sending  form 

code.bas 

2.1 

Common  settings 

Module  l.bas 

2.1 

Common  settings  and  functions 

ERSTA.mdb 

2.1 

Configuration  file 

Just  make  the 
form  running 

Before  making  changes  to  any  files  in  the  Fonns  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client  computer. 

Following  a  change  to  a  file  contained  within  the  “Forms”  project,  a  new 
ContactReport.exe  can  be  generated  as  follows: 

Step  1 .  Open  the  workspace  in  MS  Visual  Basic  v6.0  (Forms. vbp) 

Step  2.  In  the  File  Menu,  select  “Make  ContactReport.exe”. 

Step  3.  Move  the  executable  to  the  appropriate  folder  (as  described  in  Table  1-2). 


Note:  ContactReport  is  implemented  in  Visual  Basic  V6.0.  Any  changes  or  recompiling  must 
to  be  carried  out  on  a  computer  with  Visual  Basic  installed. 
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1.2.7  Building  CDTV.exe 

Table  1-9  lists  the  files  included  in  the  “CDTV”  project  for  Experiment  #3. 

Table  1-9  CDTV.exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.CPP) 
MKS  Server  (srvenglO:' 
d:\Groups\MKS\Develo] 

7001) 

pment\HFE\T AM S S  SA  TI02\process\CDTV\ 

CDTV.cpp 

2.1 

Simulates  the  image  from  a  Color  Day 
Television. 

Overlays,  cpp 

2.1 

Draws  overlays  over  the  CDTV  image. 

HEADER  FILES(.h) 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  1 

Id 

"o 

HH 

(N 

O 

HH 
l _ _ 

Ersta  communication 
types. h 

2.1 

Structure  and  function  definition  of  the 
UDP  communications. 

CDTV.h 

2.1 

Header  file  used  by  CDTV.cpp. 

cdtvglobal.h 

2.1 

Header  file  to  define  global  variables  used 
by  both  CDTV.cpp  and  Overlay. cpp. 

shmem.h 

2.1 

Shared  memory. 

udpcommM.h 

2.1 

UDP  Communication  for  Multiple  thread. 

units. h 

2.1 

Units  conversion 

PerfT finer. h 

2.1 

Sets  timer. 

LIBRARY  FILES 

MKS  Server  (srveng  10:7001)  d:\Groups\MKS\Development\HFE\TAMSS  SA  1 

T02\Library\ 

shmemMultipleThread 

2.1 

Library  file  for  shared  memory  for 
multiple  thread 

UDPM 

2.1 

Library  file  for  UDP  communication  for 
multiple  thread. 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

Before  making  changes  to  any  files  in  the  CDTV  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  Sendbox  on  a  client  computer. 

Following  a  change  to  a  file  contained  within  the  “CDTV”  project,  a  new  CDTV.exe 
can  be  generated  as  follows: 

Step  1.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (CDTV.dsw) 

Step  2.  In  the  Build  Menu,  select  “Build  CDTV.exe”  or  press  F7. 

Step  3.  The  path  of  the  executable  file  (CDTV.exe)  has  been  already  set  to 

C:\TAMSS\Experiment2\Project\TAMSS_SA\Process\CDTV\.  The  new 
executable  file  should  be  generated  in  that  folder. 
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1.2.8  Building  HLATestexe 

Table  1-10  lists  the  files  included  in  the  “HLATest”  project  for  Experiment  #3. 

Table  1-10  HLATest  .exe  Source  Code  Files 


File  Name 

Version 

Functionality 

Comments 

SOURCE  FILES  (.CPP) 

MKS  Server  (srveng  10:7001) 

d:\Groups\MKS\Development\HFE\TAMSS  SA  TI02\process\HLA\ 

HLATest.  cpp 

2.1 

Communicates  with  NTS 

ace. dll 

2.1 

Dynamic  Library  for  HLA 

HEADER  FILES(.h) 

C:\  cvsrepoYTAMSS  SA\Include\ 

Ersta  communication 
types. h 

2.1 

Structure  and  function  definition  of  the 
UDP  communications. 

shmem.h 

2.1 

Shared  memory. 

udp  commM.h 

2.1 

UDP  Communication  for  Multiple  thread. 

PerfTimer.h 

2.1 

Set  timer. 

LIBRARY  FILES 

C:\  cvsrepoYTAMSS  SA\Library\ 

shmemMultipleThread 

2.1 

Library  file  for  shared  memory  for 
multiple  thread 

UDPM.lib 

2.1 

Library  file  for  UDP  communication  for 
multiple  thread. 

timer.lib 

2.1 

Library  file  to  set  up  timer. 

Before  making  changes  to  any  files  in  the  HLA  folder,  check  out  the  whole 
workspace  and  all  files  in  the  project  (from  the  MKS  Server)  to  a  folder  on  a  client  computer. 

Following  a  change  to  a  file  contained  within  the  “HLATest”  project,  a  new 
HLSTest.exe  can  be  generated  as  follows: 

Step  1 .  Check  out  the  whole  workspace  and  all  files  in  the  project  to  a  folder. 

Step  2.  Open  the  workspace  in  MS  Visual  Studio  C++  6.0  (HLATest.dsw) 

Step  3.  In  the  Build  Menu,  select  “Build  HLATest.  exe”  or  press  F7. 

Step  4.  The  path  of  the  executable  file  (HLATest.exe)  has  been  already  set  to 

C:\TAMSS\Experiment3\Project\TAMSS_SA\Process\HLA\.  The  new 
executable  file  should  be  generated  in  that  folder. 
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1.2.9  Entity  Name  Consistency  Issues 

In  order  to  correctly  create,  update  and  destroy  entities  for  the  CDTV  image  and  Out- 
the -Window  (OTW)  scene,  entity  names  must  be  consistent  for  processes  on  both  NTS  and 
ERSTA  sides.  Refer  to  Appendix  B  “The  Structure  of  the  Entity  Name  Map  Files”  for 
instructions  on  how  to  make  the  names  consistent. 

1.2.10  Entity  Model  Consistency  Issues 

To  make  the  entities  have  the  constant  visual  image  on  both  OTW  and  the  ERSTA 
sensor  image,  the  same  entity  model  and  texture  must  be  used.  TAMSS_SA  used  the  MAK  3D 
entity  models  and  choose  the  forest  (od)  model  texture  for  both  OTW  and  ERSTA. 

1.2.11  Scenario  Set  Up 

There  are  a  total  of  four  scenarios  for  Experiment  #3.  Different  configuration  files 
must  be  set  to  run  the  four  scenarios  separately.  The  configuration  files  have  already  been 
created.  They  are  as  follows: 

•  GagetownN.adf  (N:  1-4):  Set  correct  position  and  entity  models  for  CDTV 
video. 

•  Aco_N.xml  (N:  1-4):  Set  Airspace  Co-ordinate  Order  overlays. 

•  Tactical_N.xml  (N:  1-4):  Set  tactical  overlays. 

•  ScenarioN.xml  (N:  1-4):  Set  up  process  path  and  scenario  number  for 
“Manager”  process  to  start  all  ERSTA  processes. 

On  Harry,  the  following  four  icons  appear  on  the  desktop:  SI,  S2,  S3,  S4.  To  start  a 
scenario,  just  click  on  the  corresponding  icon.  Refer  to  the  "Start-up  Procedure  for  the  ERSTA 
Mock-up"  in  Appendix  A. 

1.3  Limitations 

1.3.1  Map  Coverage 

During  Stage  1,  a  1:50K  Gagetown  Map  was  used.  The  map  covers  a  range  within 
the  following  latitudes  and  longitudes:  East:  66°  35',  West:  66°05',  South:  45°25’,  North:  45°52’. 
Based  on  the  scenario,  the  major  activities  will  be  occurring  within  this  area.  However,  the 
starting  point  and  ending  point  might  be  off  the  map.  In  this  case,  the  map  area  background  will 
be  appeared  as  grey  and  only  the  overlays  will  be  shown  correctly. 


1.3.2  Default  Autoslew 

The  default  Autoslew  function  was  implemented,  which  points  the  turrets  toward  the 
nose  of  the  aircraft,  and  is  described  in  the  HEDAD-0  [Ref  1].  If  no  position  has  been 
designated  on  the  map,  and  the  Aircraft  Heading  Slew  button  or  the  Slew  button  on  the  joystick 
is  pressed,  the  sensor  will  slew  to  the  nose  (pan  zero/  tilt  zero)  of  the  aircraft  by  default. 


Version  002 


18 


12  Apr  04 


CMC  ELECTRONICS  INC. 


HUMAN  FACTORS  ENGINEERING 


DOC  NO 
410-A64025-002 


2  INSTALLATION 

For  successful  operation,  all  files  (including  executable  files,  configuration  files  and 
data  files)  must  be  located  in  the  right  directory,  as  indicated  in  Table  1-2. 

2.1  Run  ERSTA  Software 

2.1.1  Licenses 

Install  the  following  licences: 

•  Vega  MP  Runtime  License  for  windows  with  basic  module  and  special  effects 
module  (Fx)  for  CDTV.exe 

•  VRLink  License  for  HLATest.exe 

In  the  TAMSS_ERSTA  system  for  Experiment  #3,  a  Vega  MP  Developer  license 
(including  running  license)  is  installed  on  the  computer/server  designated  as  “Harry”  (refer  to 
Figure  1-1).  The  file  CDTV.exe  runs  on  the  computer  designated  as  “Hagrid”,  to  check  out  the 
running  license  from  the  server  (Harry).  The  VR  Link  License  is  developed  and  installed  on 
Hagrid,  and  the  file  HLATest.exe  rus  on  the  same  computer. 

2.1.2  Applications 

Install  the  DMSO  RTI1.3NG  Version  6  for  HLATest.exe  application. 

In  the  TAMSS_ERSTA  system  for  Experiment  #3,  the  application  is  installed  on 
Hagrid  where  HLATest.exe  is  running. 

2.1.3  Environment 

Perform  the  following  additions/modifications  on  the  computer  operating  system 
environment  on  Hagrid: 

Step  1 .  Variable:  MAKLMGRD  LICENSE  FILE  (for  HLATest.exe) 

Value:  @machine_running_mak_license_server,  e.g.  hagrid 

Step  2.  Variable:  RTI  RID  FILE  (for  HLATest.exe) 

Value:  location  of  rid  file 

Step  3.  Variable:  Path  (Addition)  (for  HLATest.exe) 

Value:  c:\Program Files\DMSO\rtil.3ng-v6\win2000-vc6\bin 

2.2  Recompile  and  Rebuild  ERSTA  Software 

To  recompile  and  rebuild  the  ERSTA  software,  all  files  (including  Include  Files, 
Library  Files  and  Data  Files)  listed  in  Table  1-3  through  Table  1-9  must  be  located  in  the  correct 
directories  as  described  in  these  tables. 
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2.2.1  Licenses 

Install  the  following  licenses: 

Step  1 .  Vega  Developer  License  with  basic  module  and  special  effect  module  (Fx) 

for  CDTV.exe 

Step  2.  VRLink  License  for  HLATest.exe 

In  the  TAMSS_ERSTA  system  for  Experiment  #3,  a  Vega  MP  Developer  license 
(including  running  license)  is  installed  on  the  computer/server  designated  as  “Harry”  (refer  to 
Figure  1-1).  The  fde  CDTV.exe  runs  on  the  computer  designated  as  “Hagrid”,  to  check  out  the 
running  license  from  the  server  (Harry).  The  VR  Link  License  is  developed  and  installed  on 
Hagrid,  and  the  file  HLATest.exe  runs  on  the  same  computer. 

2.2.2  Applications 

Install  the  following  applications: 

Step  1.  Vega  Version  3.7.1  (ForCDTV.exe) 

Step  2.  DMSO  RTI1.3NG  Version  6  (for  HLATest.exe) 

Step  3.  MAK  VRLink  version  3.7. 1  (for  HLATest.exe) 

In  the  TAMSS_ERSTA  system  for  Experiment  #3,  Vega  application  Version  3.7.1  is 
installed  on  Hagrid,  upon  which  CDTV  was  developed.  DMSO  RTI1.3NG  Version  6  and  MAK 
VRLink  version  3.7. 1  are  installed  on  Hagrid,  upon  which  HLATest  was  developed. 

2.2.3  Environment 

Perform  the  following  additions/modifications  on  the  computer  operating  system 
environment  on  Hagrid: 

Step  1.  Variable:  Path  (Addition) 

Value:  c:\ProgramFiles\DMSO\rtil  .3ng-v6\win2000- 
vc6\bin;c:\mak\vrlink3 .7.1  -ngc\bin 

Step  2.  Variable:  RTI  BUILD  TYPE 
Value:  Win2000-vc6 

Step  3.  Variable:  RTI  HOME 

Value:  c:\Program Files\DMSO\rtil.3ng-v6 

Step  4.  Variable:  MAK  RTIDIR 

Value:  c:\Program  Files\DMSO\RTI1.3ng-v6\win2000-vc6 

Step  5.  Variable:  MAK  VRLDIR 

V alue :  c  :\mak\vrlink3 . 7 . 1  -ngc 

Hagrid  has  the  above  settings  on  the  system  environment. 
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3  OPERATION 

3.1  Startup 

Follow  the  "Start-up  Procedure  for  the  ERSTA  Mock-up"  in  Appendix  A. 

If  one  or  more  processes  are  not  available,  a  message  will  appear  in  the  command 
window.  The  TAMSS  rapid  prototype  application  should  continue  with  whatever  processes  are 
still  available,  but  functionality  for  failed  or  missing  processes  will  be  unavailable. 

3.2  Hand  Controller 

The  hand  controller  for  the  cabin  operator  is  a  bilateral  handgrip,  usable  with  either 
hand,  and  mounted  on  a  joystick.  The  hand  controller  for  the  prototype  carries  a  force  isometric 
control  thumb  controller,  several  switches,  and  two  triggers  on  the  back.  Appendix  C  “ERSTA 
Hand  Controllers”  describes  the  buttons  and  functions  on  the  Hand  Controller. 

3.3  Game  Pad 

The  game  pad  for  the  cockpit  operator  is  a  vibration  feedback  game  pad.  It  is 
convenient  for  the  operator  to  manipulate  it  on  his/her  laptop  computer.  It  has  two  analog  sticks, 
one  eight-direction  button  (used  as  four-way  for  TAMSS_SA),  nine  program  buttons  (seven  on 
the  top  and  two  on  the  side)  and  one  slider.  Appendix  C  “ERSTA  Hand  Controllers”  describes 
the  buttons  and  functions  on  the  Game  Pad. 

3.4  Map  Buttons 

There  are  nine  buttons  associated  with  the  moving  maps,  and  they  are  distributed  in 
five  columns  and  two  rows.  The  following  list  describes  each  button  and  its  functionality.  The 
buttons  are  listed  in  order  starting  with  the  right-most  column  and  moving  to  the  left-most 
column. 


OUT  -  Zoom  In  on  the  map. 

IN  -  Zoom  Out  of  the  map. 

ACO  -  Hide/Show  Airspace  Coordination  Order  (ACO)  overlays. 

TAC  -  Hide/Show  Tactical  overlays. 

ALL  -  Hide/Show  All  overlays. 

OWN  -  Hide/Show  Ownship  overlay  and  sensor  footprint. 

CLR  -  Return  Autoslew  position  to  default  (i.e.  Clear  operator 
designated  autoslew  position). 

DT  -  Hide/Show  designated  target  overlays. 

NOR  -  Indicates  current  map  orientation  (North  up).  In  Experiment  #3, 
this  button  will  toggle  the  map  orientation  between  North  Up 
(NOR)  and  Heading  up  (HDG). 

GRD  -  Turn  ON/OFF  the  map  grid. 
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3.5  Shutdown  Procedure 

Follow  the  "Shutdown  Procedure  for  the  ERSTA  Mock-up"  in  Appendix  A. 

4  CUSTOM  DATABASE 

The  database  has  two  parts:  visual  simulation  and  map.  The  visual  simulation 
database  includes  all  the  3-D  scene  databases  and  the  3-D  flight  models.  The  map  database 
includes  2-D  maps,  Airspace  Coordination  Order  data  and  2-D  symbologies. 

4.1  Add  a  New  Flight  Model 

Add  a  new  flight  model  as  follows: 

Step  1.  Add  the  new  flight  model,  including  the  Open  Flight  fde  (.fit),  Texture 
files  (.rgb)  and  the  Texture  attribute  file  (.attr),  to  the  GoodEntity  folder. 

Step  2.  Give  a  name  to  the  model,  ensuring  that  the  name  is  consistent  with  other 
files  (Refer  to  “The  Structure  of  the  Model  Map  Files”  in  Appendix  C). 

Step  3.  Add  the  model  in  the  GagetownN.adf  (N:  1-4  based  on  the  scenario)  follow 
the  VEGA  Lynx  User’s  Guide”  [Ref  3]. 

Step  4.  Make  sure  all  the  related  files  have  been  updated  with  the  new  flight 
model. 

4.2  Modify  an  ACO  Overlay 

Modifying  an  ACO  overlay  (i.e.  area,  flight  plan,  line,  symbol  or  label)  is  achieved 
through  modification  of  the  corresponding  acox.xml  file  (i.e.  acol.xml,  aco_2.xml,  etc.).  An 
area  is  defined  by  three  or  more  points,  which  correspond  to  each  “corner”  of  the  specified  area. 
A  flight  plan  is  described  by  one  or  more  locations  (i.e.  waypoints).  Each  line  is  defined  by  two 
or  more  points  (i.e.  locations)  which  are  connected  together  when  drawn.  Each  overlay  symbol 
is  described  by  its  image  (what  the  operator  will  see  on  the  map)  and  location  (where  the  symbol 
is  drawn  on  the  map).  Similarly,  each  label  is  described  by  a  text  string  (that  which  is  to  be 
displayed  on  the  map)  and  location. 

4.2.1  Modifying  an  Area 

To  modify  an  existing  area: 

Step  1 .  Locate  the  corresponding  <AREA></AREA>  pair  in  the  .xml  file. 

Step  2.  Change  the  location  of  a  corner  on  the  map  by  updating  the  numbers 
between  the  <LOCATION>  and  </LOCATION>  tags  for  that  given 
comer. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  the  corner  in 
UTM  co-ordinates. 
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Step  3.  Remove  an  existing  corner  by  deleting  the  corresponding 

<LOCATION></LOCATION>  tags  and  everything  between  them. 

Step  4.  Add  a  new  comer  by  inserting  a  new  <LOCATION></LOCATION>  pair 
within  the  corresponding  area  and  specifying  the  X,  Y,  and  Z  values  as 
described  above. 


WARNING:  The  order  in  which  comers  are  specified  is  important.  Specify  each  comer  in  the 
order  you  would  encounter  them  if  walking  clockwise  or  counter  clockwise  around  the  entire 
perimeter  of  the  area  being  defined.  Failure  to  do  so  may  cause  the  area  to  be  drawn  incorrectly. 


4.2.2  Modifying  a  Flight  Plan 

To  modify  an  existing  flight  plan: 

Step  1 .  Locate  the  corresponding  <FLIGHTPLAN></FLIGHTPLAN  >  pair  in  the 
.xml  file. 

Step  2.  Change  the  location  of  a  waypoint  on  the  map  by  updating  the  numbers 
between  the  <LOCATION>  and  </LOCATION>  tags  for  that  given 
waypoint. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  the  corner  in 
UTM  coordinates. 

Step  3.  Remove  an  existing  waypoint  by  deleting  the  corresponding 

<LOCATION></LOCATION>  tags  and  everything  between  them. 

Step  4.  Add  a  new  waypoint  by  inserting  a  new  <LOCATION></LOCATION> 
pair  within  the  corresponding  flight  plan  and  specifying  the  X,  Y,  and  Z 
values  as  described  above. 

4.2.3  Modifying  a  Line 

To  modify  an  existing  line: 

Step  1 .  Locate  the  corresponding  <LINE></LINE  >  pair  in  the  .xml  file. 

Step  2.  Change  the  location  of  a  line  point  by  updating  the  numbers  between  the 
<LOCATION>  and  </LOCATION>  tags  for  that  given  point. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  that  point  in 
UTM  coordinates. 

Step  3.  Remove  an  existing  line  point  by  deleting  the  corresponding 

<LOCATION></LOCATION>  tags  and  everything  between  them. 

Step  4.  Add  a  new  line  point  by  inserting  a  new  <LOCATION></LOCATION> 
pair  within  the  corresponding  line  and  specifying  the  X,  Y,  and  Z  values 
as  described  above. 
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4.2.4  Modifying  a  Symbol 

To  modify  an  existing  symbol: 

Step  1.  Locate  the  corresponding  <SYMBOL></SYMBOL>  pair  in  the  .xml  file. 

Step  2.  Display  a  different  image  on  the  map  by  updating  the  file  name  that 
precedes  the  end  tag  </IMAGE>. 

Step  3.  Change  the  location  of  the  symbol  on  the  map  by  updating  the  numbers 
between  the  <LOCATION>  and  </LOCATION>  tags. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  the  symbol  in 
UTM  coordinates. 

4.2.5  Modifying  a  Label 

To  modify  an  existing  label: 

Step  1 .  Locate  the  corresponding  <LABEL></LABEL>  pair  in  the  .xml  file. 

Step  2.  Change  the  text  string  being  displayed  on  the  map  by  updating  the  text 
string  between  the  <TEXT></TEXT>  tags. 

Step  3.  Change  the  location  of  the  text  string  on  the  map  by  updating  the  numbers 
between  the  <LOCATION>  and  </LOCATION>  tags. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  the  label  in  UTM 
co-ordinates. 

4.2.6  Adding  a  New  Area  or  Deleting  an  Existing  Area 

To  add  a  new  area: 

Step  1 .  Insert  a  new  <AREA></AREA>  pair  between  the  <OVERLAY>  and 
</OVERLAY>  tags  (outside  of  any  existing  <AREA></AREA>, 
<FLIGHTPLAN></FLIGHTPLAN>,  <LINE></LINE>, 
<LABEL></LABEL>  and  <SYMBOL></SYMBOL>  pairs). 

Step  2.  Define  at  least  three  comers  and  specify  the  corresponding  LOCATION 
data  as  described  above. 

To  delete  an  existing  area,  delete  the  corresponding  <AREA></AREA>  tags  and 
everything  between  them. 
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4.2.7  Adding  a  New  Flight  Plan  or  Deleting  an  Existing  Flight  Plan 

To  add  a  new  flight  plan: 

Step  1 .  Insert  a  new  <FLIGHTPLAN></FLIGHTPLAN>  pair  between  the 

<OVERLAY>  and  </OVERLAY>  tags  (outside  of  any  existing 
<AREAx/AREA>,  <FLIGHTPLAN></FLIGHTPLAN>, 
<LINE></LINE>,  <LABEL></LABEL>  and  <SYMBOL></SYMBOL> 
pairs). 

Step  2.  Define  at  least  one  waypoint  and  specify  the  corresponding  LOCATION 
data  as  described  above. 

To  delete  an  existing  flight  plan,  delete  the  corresponding 
<FLIGHTPLAN></FLIGHTPLAN>  tags  and  everything  between  them. 

4.2.8  Adding  a  New  Line  or  Deleting  and  Existing  Line 

To  add  a  new  line: 

Step  1 .  Insert  a  new  <LINE></LINE>  pair  between  the  <OVERLAY>  and 

</OVERLAY>  tags  (outside  of  any  existing  <AREA></AREA>, 
<FLIGHTPLAN></FLIGHTPLAN>,  <LINE></LINE>, 
<LABEL></LABEL>  and  <SYMBOL></SYMBOL>  pairs). 

Step  2.  Define  at  least  two  points  and  specify  the  corresponding  LOCATION  data 
as  described  above. 

To  delete  an  existing  line,  delete  the  corresponding  <LINE></LINE>  tags  and 
everything  between  them. 

4.2.9  Adding  a  New  Symbol  or  Deleting  an  Existing  Symbol 

To  add  a  new  symbol: 

Step  1 .  Insert  a  new  <SYMBOL></SYMBOL>  pair  between  the  <OVERLAY> 

and  </OVERLAY>  tags  (outside  of  any  existing  <AREA></AREA>, 
<FLIGHTPLAN></FLIGHTPLAN>,  <LINE></LINE>, 
<LABEL></LABEL>  and  <SYMBOL></SYMBOL>  pairs). 

Step  2.  Specify  the  corresponding  IMAGE  and  LOCATION  data  as  described 
above. 

To  delete  an  existing  symbol,  delete  the  corresponding  <SYMBOL></SYMBOL> 
tags  and  everything  between  them. 
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4.2.10  Adding  a  New  Label  or  Deleting  and  Existing  Label 

To  add  a  new  label: 

Step  1 .  Insert  a  new  <LABEL></LABEL>  pair  between  the  <OVERLAY>  and 
</OVERLAY>  tags  (outside  of  any  existing  <AREA></AREA>, 
<FLIGHTPLAN></FLIGHTPLAN>,  <LINE></LINE>, 
<LABEL></LABEL>  and  <SYMBOL></SYMBOL>  pairs). 

Step  2.  Specify  the  corresponding  TEXT  and  LOCATION  data  as  described 
above. 

To  delete  an  existing  label,  delete  the  corresponding  <LABEL></LABEL>  tags  and 
everything  between  them. 

4.3  Modify  a  Tactical  Overlay 

Modifying  a  tactical  overlay  (i.e.  symbol  or  label)  is  achieved  through  modification  of 
the  corresponding  tactical_N.xml  file  (N:l-4  -  i.e.  tactical  l.xml,  tactical  _2.xml,  etc.).  Each 
overlay  symbol  is  described  by  its  image  (what  the  operator  will  see  on  the  map)  and  location 
(where  the  symbol  is  drawn  on  the  map).  Similarly,  each  label  is  described  by  a  text  string  (that 
which  is  to  be  displayed  on  the  map)  and  location. 

4.3.1  Modifying  an  Existing  Symbol 

To  modify  an  existing  symbol: 

Step  1.  Locate  the  corresponding  <SYMBOL></SYMBOL>  pair  in  the  .xml  file. 

Step  2.  Display  a  different  image  on  the  map  by  updating  the  file  name  that 

precedes  the  end  tag  </IMAGE>.  To  change  the  location  of  the  symbol  on 
the  map,  update  the  numbers  between  the  <LOCATION>  and 
</LOCATION>  tags. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  the  symbol  in 
UTM  co-ordinates. 

4.3.2  Modifying  an  Existing  Label 

To  modify  an  existing  label: 

Step  1 .  Locate  the  corresponding  <LABEL></LABEL>  pair  in  the  .xml  file. 

Step  2.  Change  the  text  string  being  displayed  on  the  map  by  updating  the  text 
string  between  the  <TEXT></TEXT>  tags. 

Step  3.  Change  the  location  of  the  text  string  on  the  map  by  updating  the  numbers 
between  the  <LOCATION>  and  </LOCATION>  tags. 

These  three  numbers  represent  the  X,  Y,  and  Z  values  of  the  label  in  UTM 
co-ordinates. 
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4.3.3  Adding  a  New  Symbol  or  Deleting  an  Existing  Symbol 

To  add  a  new  symbol: 

Step  1 .  Insert  a  new  <SYMBOL></SYMBOL>  pair  between  the  <OVERLAY> 
and  </OVERLAY>  tags  (outside  of  any  existing  <LABEL></LABEL> 
and  <SYMBOL></SYMBOL>  pairs). 

Step  2.  Specify  the  corresponding  IMAGE  and  LOCATION  data  as  described 
above. 

To  delete  an  existing  symbol,  delete  the  corresponding  <SYMBOL></SYMBOL> 
tags  and  everything  between  them. 

4.3.4  Adding  a  New  Label  or  Deleting  an  Existing  Symbol 

To  add  a  new  label: 

Step  1 .  Insert  a  new  <LABEL></LABEL>  pair  between  the  <OVERLAY>  and 
</OVERLAY>  tags  (outside  of  any  existing  <LABEL></LABEL>  and 
<SYMBOL></SYMBOL>  pairs). 

Step  2.  Specify  the  corresponding  TEXT  and  LOCATION  data  as  described 
above. 

To  delete  an  existing  label,  delete  the  corresponding  <LABEL></LABEL>  tags  and 
everything  between  them. 
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6  ACRONYMS  AND  ABBREVIATIONS 


ACO 

ARP 

Airspace  Coordination  Order 

Aircraft  Reference  Position 

CAS 

CDTV 

Close  Air  Support 

Colour  Day  Television 

ERSTA 

Electro-Optic  Reconnaissance,  Surveillance  and  Target  Acquisition 

GUI 

Graphical  User  Interface 

HLA 

High  Level  Architecture 

I/O 

Input/Output 

NTS 

Networked  Tactical  Simulator 

OTW 

Out-the- Window 

RTI 

Run  Time  Infrastructure 

SA 

Situational  Awareness 

TAMSS 

TDP 

TI 

Tactical  Aviation  Mission  System  Simulation 

Technology  Demonstration  Project 

Thermal  Imager 

UDP 

UTM 

User  Datagram  Protocol 

Universal  Transverse  Mercator 

XML 

Extensible  Markup  Language 
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Appendix  A 

APPENDIX  A  -  START-UP  PROCEDURE  FOR  THE  ERSTA 
MOCK-UP  (EXPERIMENT  #3) 

This  section  describes  the  ERSTA  Mock-Up  equipment  and  processes,  how  to 
prepare  the  equipment  and  start  the  applications  required  to  perform  Experiment  #3,  and  how  to 
stop  the  applications  and  shut  down  the  system. 

A.l  Overview 

The  ERSTA  Mock-Up  has  a  total  of  thirteen  processes  running  on  the  three 
computers  as  shown  in  Table  A-l : 

Table  A-l:  Processes  and  Computers  for  ERSTA 


Computer 

Process 

Harry 

.  Sensor:  Sensor  Simulation 

.  CDTV_Video.exe:  Grabs  CDTV  Video  Signal  from  Hagrid 

.  ERSTAForms.exe:  ERSTA  Forms 

.  Map.exe:  Tactical  Map 

.  Launcher.exe:  Listening  to  start  the  processes  on  Harry 
.  Manager.exe:  Start  the  ERSTA  Simulation 

Hagrid 

.  CDTV.exe:  CDTV  Video  simulation 

.  HLAtest.exe:  HLA 

.  runlm.exe:  Run  VR  LINK  License  Manager  for  HLA 
.  lmdown.exe:  Shutdown  VR  LINK  License  Manager 
.  Launcher.exe:  Listening  to  start  the  processes  on  Hagrid 

Hermione 

.  Map.exe:  Cockpit  Map 

.  Launcher.exe:  Listening  to  start  the  processes  on  Hermione 

A.2  Preparing  and  Starting  Up  the  Equipment 

Step  1 .  Verify  the  joystick  and  game  pad  connection. 

-  Make  sure  the  joystick  is  connected  to  the  COM2  port  on  Harry. 

-  Make  sure  the  game  pad  is  connected  to  one  of  the  USB  ports  on  Harry. 
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Step  2.  Verify  the  network  connections 

-  There  should  be  four  network  cables  connected  to  the  hub:  one  from  the 
NTS  network  hub  and  the  other  three  from  the  network  cards  on  Harry, 
Hagrid  and  Hermione. 

-  Verfiy  that  the  indicate  lights  on  the  hub  for  these  four  connections  are 
bright  and  flashing. 

Step  3.  Verify  that  the  video  connections  are  as  follows: 

-  The  splitter  input  is  connected  to  the  S-Video  output  on  Hagrid. 

-  One  splitter  output  is  connected  to  the  ATI  video  card  input  on  Harry. 

-  The  other  splitter  output  is  connected  to  the  ATI  video  card  input  on 
Hermione. 

Step  4.  Turn  on  the  three  computers  in  the  following  order:  Harry,  Hermione, 
Hagrid 

Note:  To  ensure  that  the  TV  Channel  stays  ON  on  Hagrid  for  the  video 
capture,  Hagrid  must  start  after  Harry  and  Hermione. 

Step  5.  Log  in  to  all  the  computers  as  “Exp3”  (no  password). 

The  Launcher.exe  has  been  added  to  the  start  menu.  It  will  be  started 
automatically  once  the  computers  have  been  logged  in  to  as  “Exp3”.  A 
command  window  will  appear  and  will  stop  printing  at  “Listening. . .” 

Note:  If  the  Launcher  command  window  is  closed  by  mistake,  double¬ 
click  on  the  “Launcher”  icon  on  the  desktop  of  the  computer  to  run  it 
again. 

Step  6.  Verify  the  TV  Channel  and  Display  Resolution  on  Hagrid 

-  Make  sure  the  TV  channel  on  Hagrid  is  ON  by  verifying  that  the  TV 
channel  is  highlighted  in  green  under  Control  Panel->Display->Setting- 
>Advanced->  Displays.  If  it  is  highlighted  in  red,  click  on  the  channel 
button  on  the  left  to  make  it  green. 

-  The  display  resolution  on  Hagrid  should  be  640*480. 

Note:  To  change  the  display  resolution,  go  to  Control  Panel->Display- 
>Settings,  move  the  Screen  area  slider  left<->right  to  the  correct 
resolution. 

Step  7.  Verify  the  Rotation  and  Resolution  of  the  Tactical  Map  Display  on  Harry 

-  The  Tactical  Map  Display  (right  side  display  of  the  ERST  A  station) 
must  be  in  Landscape  orientation.  If  it  is  not,  right  click  anywhere  on 
the  background  of  the  display  and  select  “Rotate”. 

-  The  resolution  of  the  Tactical  Map  Display  must  be  1024*768. 
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Step  8.  Verify  the  Rotation  and  Resolution  of  the  Video  Display  on  Harry 

-  The  Video  Display  (left  side  display  of  the  ERSTA  station)  must  be  in 
Portrait  orientation.  If  not,  right  click  on  anywhere  of  the  background 
of  the  display  and  select  “Rotate”. 

-  The  resolution  of  the  Video  Display  must  be  768*1024  (shown  as 
1024*768  in  display  settings). 

Step  9.  Ensure  that  the  resolution  of  the  Cockpit  Map  Display  on  Hennione  is 
800*600 

A.3  Starting  the  Applications 

Step  1 .  Start  the  applications 

-  Click  on  the  “runLm”  icon  on  Hagrid  and  wait  for  a  few  moments  until 
the  following  lines  appear: 

“hvl2  dvll  dvl2” 

“vl3” 

Note:  This  process  only  needs  to  be  started  once.  It  can  be  kept  running 
until  the  experiments  are  finished. 

Step  2.  There  are  four  icons  on  the  desktop  of  Harry  (SI,  S2,  S3,  S4),  which  are 

used  to  start  the  four  scenarios  separately.  Double-click  on  one  of  the  four 
icons  to  start  the  corresponding  scenario. 

•  Tactical  Map  will  appear  on  the  Tactical  Display  with  overlays  of  the 
corresponding  scenario. 

•  Cockpit  Map  will  appear  on  the  Cockpit  Display  with  overlays  of  the 
corresponding  scenario. 

•  The  CDTV  video  image  will  appear  at  the  top  half  part  of  the  video 
display.  The  image  starts  as  white  and  it  takes  about  three  to  five 
minutes  before  the  scene  shows  up. 

•  The  same  CDTV  video  image  will  also  appear  in  the  TV  frame  of  the 
cockpit  center  display.  The  image  starts  as  white  and  it  takes  about 
three  to  five  minutes  before  the  scene  shows  up. 

•  The  CDTV  video  image  should  move  with  the  joystick  and  game  pad 
(to  make  sure  the  sensor  is  running). 

•  The  CDTV  video  image  should  move  with  the  OTW  scene  (to  make 
sure  HLA  is  running). 

•  The  ERSTA  fonn  frame  will  appear  at  the  bottom  half  part  of  the  video 
display. 

•  Click  the  Target  Mark  button  on  the  joystick  or  game  pad.  The  contact 
form  pops  up  within  the  form  frame.  Click  on  Cancel  to  make  it 
disappear. 
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•  On  Hagrid,  the  VR  Link  licenses  must  be  checked  out.  The  lines  below 
will  be  printed  out  on  the  runLm  Command  window. 

“OUT  :  “hvll”  Exp3@Hagrid 
“OUT  :  “hvl2”  Exp3@Hagrid 

Note:  If  the  license  is  not  checked  out  correctly,  HLA  will  NOT  run 
properly.  Refer  to  the  section  below  “Run  and  Shutdown  the  VR  Link 
License  Manager”  to  solve  the  problem. 

The  ERSTA  Mock-Up  is  now  ready. 

A.4  Shutting  Down  the  Applications 

Step  1.  Stop  the  applications  as  follows: 

-  Make  sure  the  Stage  RTI  is  still  running  and  is  not  frozen  or  paused. 

—  Press  “q”  on  the  keyboard  on  Hagrid  (the  command  syntax  is  not  case 
sensitive) 

Step  2.  Verify  that  all  processes  have  been  closed. 

On  Harry: 

-  Tactical  Map  window  should  be  closed. 

-  Video  Capture  window  should  be  closed. 

-  ERSTA  Form  window  should  be  closed. 

-  In  the  Launcher  command  window,  “Sensor  has  been  closed!”  should 
be  displayed. 

On  Hagrid: 

-  CDTV  window  should  be  closed. 

-  In  the  Launcher  command  window,  “HLA  has  been  shut  down!”  should 
be  displayed. 

-  The  VRlink  licenses  should  be  checked  back  in.  In  the  runLm 
command  window,  the  lines  below  should  be  printed  out. 

IN:  “hvll”  Exp3@Hagrid” 

IN:  “hvl2”  Exp3@Hagrid” 

Note:  If  the  above  message  is  not  printed  out,  the  license  is  not  checked 
back  in,  HLA  can  NOT  restart  properly.  Refer  to  the  section  below 
“Run  and  Shutdown  the  VR  Link  License  Manager”  to  solve  the 
problem. 


Version  002 


A.4 


12  Apr  04 


CMC  ELECTRONICS  INC. 


HUMAN  FACTORS  ENGINEERING 

On  Hermione: 

-  Cockpit  Map  window  should  be  closed. 

The  ERSTA  system  is  now  shut  down  and  ready  to  restart. 
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A.5  Shutting  Down  the  Applications  Individually 

If  the  ERSTA  Mock-Up  cannot  be  shut  down  properly  as  a  result  of  errors  that  may 
have  occurred  while  following  the  procedures  described  in  "Stopping  the  Applications",  all  the 
processes  included  in  the  ERSTA  Mock-Up  must  be  shut  down  individually,  as  follows: 

On  Harry: 

Step  1 .  Shut  down  Tactical  Map  by  moving  the  mouse  to  the  Tactical  Map 

window  and  clicking  on  it  to  activate  the  Tactical  Map  window  ,  then 
press  the  “Esc”  key  on  the  keyboard. 

Step  2.  Shut  down  Video  Capture  by  moving  the  mouse  to  Video  Capture  window 
and  clicking  on  it  to  activate  the  Video  Capture  window,  then  press  the 
“Esc”  key  on  the  keyboard. 

Step  3.  Shut  down  ERSTA  Forms  by  clicking  on  “SYSTEM”  on  the  ERSTA 
Form  menu  (at  bottom  right),  then  clicking  on  “LOGOUT”  from  the 
dropdown  menu. 

Step  4.  Shut  down  Sensor  Simulation  by  closing  the  Launcher  command  window. 

To  do  this,  click  on  the  close  icon  (x)  at  the  top-right  side  of  the  window. 

Step  5.  Rerun  the  launcher  window  by  clicking  the  “Launcher”  icon  on  the 
desktop  to  make  it  ready  to  restart  ERSTA. 

On  Hagrid: 

Step  1 .  Shut  down  CDTV  and  HLA  by  closing  the  Launcher  command  window. 

To  do  this,  click  on  the  close  icon  (x)  at  the  top-right  side  of  the  window. 

Step  2.  Rerun  the  launcher  window  by  clicking  the  “Launcher”  icon  on  the 
desktop  to  make  it  ready  to  restart  ERSTA. 

Step  3.  Verify  that  the  VR  link  license  is  checked  back  in  by  verifying  that  the 
following  lines  are  displayed  in  the  runLm  command  window: 

IN:  “hvll”  Exp3@Hagrid” 

IN:  “hvl2”  Exp3@Hagrid” 

If  the  lines  are  not  displayed,  refer  to  the  section  below  “Run  and 
Shutdown  the  VR  Link  License  Manager”  to  solve  the  problem. 
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On  Hermione: 

Step  1 .  Shut  down  the  Cockpit  Map  by  moving  the  mouse  to  Cockpit  Map 
window  and  clicking  on  it  to  make  the  Cockpit  Map  window  active. 

Step  2.  Press  the  “Esc”  key  on  the  keyboard. 

A.6  Run  and  Shut  Down  VRLink  License  Manager 

The  VR  Link  license  is  installed  to  run  HLA.  When  the  ERSTA  system  starts  up,  the 
license  will  be  checked  out,  and  when  the  ERSTA  system  shuts  down,  the  license  will  be 
checked  back  in  so  that  it  can  be  reused  again. 

If  the  license  does  not  check  out  and  check  in  correctly,  HLA  will  not  run  properly. 

Step  1 .  Run  VR  Link  License 

-  Click  runLm  on  the  desktop  to  activate  the  runLm  command  window.  It 
takes  about  one  minute  to  make  the  license  ready  and  for  the  following 
lines  to  be  displayed. 

“hvl2  dvll  dvl2” 

“vl3” 

Step  2.  VR_Link  License  Check  Out 

Step  3.  When  the  ERSTA  system  (HLA)  is  started,  the  license  will  be  checked 
out.  After  a  few  minutes,  the  following  lines  are  displayed: 

OUT:  “hvll”  Exp3@Hagrid” 

OUT:  “hvl2”  Exp3@Hagrid” 

If  these  lines  are  not  displayed,  verify  /  perform  the  following: 

-  Make  sure  Stage  RTI  is  running  properly  without  being  frozen,  paused 
or  shut  down. 

-  Shut  down  ERSTA  individually  (refer  to  Shutting  Down  the 
Applications  Individually). 

-  Close  the  runLm  command  window  (refer  to  Forcing  the  License 
Manager  (LM)  to  Close). 

-  Restart  ERSTA  (refer  to  Starting  the  Applications). 
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Step  4.  VRLink  License  Check  In 

When  the  ERAST  system  (HLA)  is  shut  down,  the  license  will  be  checked 
back  in.  After  a  few  minutes,  the  following  lines  are  displayed: 

IN:  “hvll”  Exp3@Hagrid” 

IN:  “hvl2”  Exp3@Hagrid” 

If  these  lines  are  not  displayed,  verify  that  the  Stage  RTI  is  running 
properly  without  being  frozen,  paused  or  shut  down. 

Step  5.  Forcing  the  License  Manager  (LM)  check  back  in: 

Close  the  Launcher  Command  window  to  force  HLA  process  shut  down. 
Doing  this  will  make  the  VRlink  license  checked  back  in.  Make  sure  to 
rerun  the  launcher  window  by  clicking  the  “Launcher”  icon  on  the  desktop 
to  make  it  ready  to  restart  ERSTA. 

Step  6.  Forcing  the  License  Manager  (LM)  to  Close: 

Run  “hndown”  on  the  desktop.  Doing  this  will  force  the  license  to  check 
back  in  and  the  “runlrn”  command  window  to  close.  Make  sure  to  rerun 
the  the  license  manager  by  clicking  the  “runlrn”  icon  on  the  desktop  to 
make  it  ready  to  restart  ERSTA. 
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APPENDIX  B  -  THE  STRUCTURE  OF  THE  MODEL  MAP 
FILES  FOR  SCENARIO  SETUP 


Figure  B-l  shows  the  structure  of  the  model  map  files  on  both  NTS  and  ERSTA  side. 


Network 


NTS 


Figure  B-l  Structure  of  the  Model  Map  Files 


There  are  a  total  of  five  files  on  map  entities  for  the  NTS  and  ERSTA  applications. 
The  file  names,  locations  and  functions  are  described  in  Table  B-l. 

Table  B-l  Model  Map  File  Names,  Locations,  Functions,  Formats  and  Examples 


File  Name 

Location 

Function 

Format 

Example 

ntsotwladb. 

adf 

CURSE-EOS-> 

N:\data\ADF\ 

Application  Define  File 
used  by  Out  The  Window 
(OTW)  application 
nts_otw.exe)  to  set  up  the 
system  variables  including 
the  paths  to  access  the 

FLT  format  models 

Set  up  through 
Lynx  (Graphic 
User  Interface  of 
Vega) 

[Ref  4] 

entity,  map 

CURSE-EOS-> 
C:\tamss  rfidata 
\hla\ini\ 

Entity  map  file  to  map 
entity  names  between 
STAGE 

(tamss  stage  sim.exe) 
and  OTW  (nts  otw.exe) 

STAGE  name  is 
on  the  left  and 
the  Vega  name  is 
on  the  right. 

[Ref  5] 
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dis_eg_type.txt 

CURSE-EOS-> 

C:\vpi\stage400 

\data\ 

Distribute  Type  File  to 
define  the  entity  names. 
Used  by  Stage 
(tamss  stage  sim.exe) 
and  HLA 

(tamss_stage_de.exe)  in 
the  NTS  side. 

There  are  six 
columns.  Refer 
to  Example  file 
for  the 

definition.  DIS 
Type  (the  first 
column)  needs  to 
be  unique 
for  each  entity. 

[Ref  6] 

Gagetownl.adf 

Gagetown2.adf 

Gagetown3.adf 

Gagetown4.adf 

Hagrid-> 

C:\TAMSS\Exp 

eriment3\Projec 

t\Process\SysM 

ag\Launcher\ 

Application  Define  File 
used  by  CDTV  video 
image  application 
(CDTV.exe)  to  set  up  the 
system  variables  including 
the  paths  to  access  the 

FLT  format  models 

Set  up  through 
Lynx  (Graphic 
User  Interface  of 
Vega) 

[Ref  4] 

dis_eg_type 

Hagrid-> 

C:\TAMSS\Exp 

eriment3\Projec 

t\Process\SysM 

ag\Lanucher\ 

Distribute  Type  File  to 
define  the  entity  names. 
Used  by  CDTV 
(CDTV.exe)  and  HLA 
(tamss_stage_de.exe)  on 
the  ERSTA  side 

Should  be  the 
same  file  as 
dis_eg_type.txt 
on  the  NTS  side. 
Note:  No 
extension  name 
(.txt)  in  the 
ERSTA  side. 

[Ref  6] 

Table  B-2  shows  the  entities  and  scene  tiles  for  the  four  Scenarios.  The  consistency 
of  the  entity  names  is  important.  The  names  are  case  sensitive.  The  five  files  listed  in  the  Table 
B-l  must  use  exactly  the  same  entity  names  provided  in  Table  B-2. 

Table  B-2  Entity  Names  and  Scene  Tiles  for  the  Scenarios 


Item 

Scenario  1 

Scenario  2 

Scenario  3 

Scenario  4 

Models 

Ml  13  APC 

MIA2  Abra 

M110  SPH 

Ml 045  Tow 

AH  64 

M939A2 

MI-28  Havoc 

SU-25  Frogfoot 

BMP-2 

BRDM-2-AT-5 
(BTR  80) 

T-72 

ZSU-23-4 

Ml 025  Hum  Tow 

Ml 025  Hum  Tow 

Ml 025  Hum  Tow 

Ml 025  Hum  Tow 

M109 

M109 

M270  MLRS 

M270  MLRS 

Tents 

Tents 

Tents 

Tents 

M2A3  Brad 

M2A3  Brad 

Hum  Avenger 

M2A3  Brad 

Hum  Avenger 

Hum  Avenger 

CHI  46 

Ml  13  APC 

AH64  Crashed 

Scene 

Gagetown  0  1 

Gagetown  1  1 

Gagetown  2  0 

Gagetown  0  2 

Gagetown  0  2 

Gagetown  1  2 

Gagetown  2  1 

Gagetown  0  3 

Gagetown  1  1 

Gagetown  1  3 

Gagetown  2  2 

Gagetown  0  4 
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Item 

Scenario  1 

Scenario  2 

Scenario  3 

Scenario  4 

Gagetown  1  2 

Gagetown  2  1 

Gagetown  3  0 

Gagetown  1  2 

Gagetown2  1 

Gagetown  2  2 

Gagetown  3  1 

Gagetown  1  3 

Gagetown  2_2 

Gagetown  2_3 

Gagetown  3_2 

Gagetown  1_4 
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APPENDIX  C  -  ERSTA  HAND  CONTROLLERS 


C.l  Joystick 

Figure  C-l  illustrates  the  switch  definition  of  the  Joystick  and  button  functions  that 
are  defined  in  Table  C-l .  The  four-way  button  on  the  right  side  is  reserved  for  TI 
sensor  control,  which  is  not  available  for  Experiment  #3. 


Laser 

Range 

Finder 


Target 

Track 


Map 

Slew 

Pos 

Track 


Zoom  In 


Zoom  Out 


Figure  C-l  ERSTA  Joystick 


Laser 

Designate 

Target 

Mark 
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C.2  Wing  Man  Game  Pad 

Figure  C-2  illustrates  the  switch  definition  of  the  Wing  Man  Game  Pad  and  button 
functions  that  are  defined  in  Table  C-l. 


When  designate,  both 
Designate  button  and  Designate 
Protection  button  must  be 
pushed 

Designate 


Target  Mark 


Map 

Slew 


Target 

Track 


Designate 

Protection 


Zoom  In 


Y  Axis 

Zoom  Out 


Figure  C-2  ERSTA  Wing  Man  Game  Pad 
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Table  C-l  Hand  Controller  and  Button  Functions 


Function 

Type 

Location 

Description 

Map 

Slew 

Momentary 

Switch 

Joystick,  Left, 

4  position 
button,  Up 

The  button  commands  the  sensor  slew  to  the  position 
designated  on  either  the  tactical  map  or  cockpit  map. 

If  there  is  no  position  designated  on  the  map,  the 
sensor  slews  to  the  default  auto-slew  position 
pointing  to  the  nose  (pan  zero/  tilt  zero)  of  the 
aircraft. 

Slew  will  start  once  the  button  is  pressed  and  stop 
when  the  sensor  reaches  the  designated  position  or 
within  five  seconds,  whichever  comes  first. 

Target 

Track 

Momentary 

Switch 

Joystick,  Left, 

4  position 
button,  Left 

The  button  commands  the  sensor  to  track  a  dynamic 
object  within  the  track  window. 

The  target  auto-track  starts  when  the  button  is  pressed 
and  stops  when  the  joystick  is  moved  from  the  zero 
position  on  either  the  X  or  Y  axis. 

Position 

Track 

Momentary 

Switch 

Joystick,  Left, 

4  position 
button,  Right 

The  button  commands  the  sensor  to  track  a  still  object 
or  position  at  the  centre  of  the  window. 

The  position  auto-track  starts  when  the  button  is 
pressed  and  stops  when  the  joystick  is  moved  from 
the  zero  position  on  either  the  X  or  Y  axis. 

Zoom  in 

Momentary 

Switch 

Joystick, 
Middle,  4 
position 
button,  Up 

When  the  button  is  pressed  the  zoom  factor  increases. 
Holding  the  button  provides  continuous  zoom  to  the 
system’s  maximum  zoom  (20) 

Zoom  in 

Momentary 

Switch 

Joystick, 
Middle,  4 
position 
button,  Down 

When  this  button  is  pressed,  the  zoom  factor 
decreases.  Holding  the  button  provides  continuous 
zoom  to  the  system's  minimum  zoom  (1) 

Laser 

Range 

Finder 

(LRF) 

Momentary 

Switch 

Joystick, 

Front,  red 
button 

Pressing  and  releasing  the  button  toggles  the  Laser 
Range  Finder  OFF  and  ON.  The  default  LRF  is  ON; 
that  is,  LRF  is  ON  when  the  system  starts. 

Laser 

Designate 

Firing 

(LDF) 

Momentary 

Switch 

Joystick, 

Back,  up  with 
guarded  bar 

Pressing  the  trigger  starts  the  Laser  Designate  Firing 
(LDF).  Holding  the  trigger  continues  the  operation  of 
the  LDF  and  releasing  the  trigger  stops  the  LDF. 
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Function 

Type 

Location 

Description 

Target 

Mark 

Momentary 

Switch 

Joystick, 

Back,  down 

Pressing  the  trigger  results  in  a  Contact  Form  popping 
up  on  the  bottom  half  of  the  visual  simulation  display. 
Once  the  form  is  sent,  the  target  position  is  marked 
on  the  map. 

The  trigger  also  starts  position  auto-track.  The  auto¬ 
track  stops  if  a  “Cancel”  button  on  the  form  is  clicked 
or  if  the  joystick  is  moved  from  the  zero  position  on 
either  X  or  Y  axis. 

C.3  Cockpit  and  Cabin  Hand  Controller  Design 

Criteria  for  the  Current  Turret  Controller  is  as  follows: 

•  The  first  operator  to  deflect  his  turret  steering  control  from  its  centre  (dead- 
zone)  position  will  become  the  Current  Turret  Controller.  Once  the  Current 
Turret  Controller  is  established,  the  other  operator  will  not  have  access  to 
steering  and  mode  buttons  (Slew,  position  auto-track  and  target  auto-track). 
Refer  to  Table  1.  However,  the  other  operator  will  still  have  access  to  the 
button  controls  for  non-turret  related  functions.  As  soon  as  the  Current  Turret 
Controller  lets  the  joystick  return  to  its  centre  position,  both  operators  have 
access  again. 

•  The  first  operator  to  select  a  mode  button  will  become  the  Current  Turret 
Controller  as  long  as  that  mode  is  still  active. 

•  The  CDTV  zoom  control  will  work  in  the  same  way.  The  first  operator  to 
press  the  zoom  button  will  became  the  Current  Zoom  Controller.  Once  the 
Current  Zoom  Controller  is  controlling  the  zoom,  the  other  operator  cannot 
control  the  zoom  anymore  until  the  Current  Zoom  Controller  releases  the 
zoom  button  and  it  returns  to  its  centre  position. 

•  The  Current  Turret  Controller  and  the  Current  Zoom  Controller  could  be 
different  operators. 
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•  When  the  map  is  clicked  and  the  target  position  field  on  the  CAS  form  does 
not  have  focus,  the  map  click  will  set  the  map-slew  position.  There  is  one 
map-slew  position  stored  for  each  map;  they  are  independent  of  one  another. 
When  the  slew  button  is  pressed,  the  turret  will  slew  to  the  map  position 
associated  with  the  operator;  that  is,  if  the  Cockpit  Operator  clicked  a  position 
on  the  Cockpit  Map,  the  turret  only  slews  to  that  position  if  the  Cockpit 
Operator  presses  the  slew  button.  If  the  Cabin  Operator  presses  the  slew 
button,  then  turret  will  only  slew  to  the  position  on  Tactical  Map  he  selected. 
If  the  Cockpit  Operator  selects  position  on  the  Cockpit  Map  and  the  Cabin 
Operator  selects  nothing,  when  the  Cockpit  operator  presses  the  slew  button 
on  the  Cockpit  joystick,  the  turret  will  slew  to  the  selected  position  on  the 
Cockpit  Map.  When  the  Cabin  Operator  presses  the  slew  button  on  the  Cabin 
joystick,  the  turret  will  slew  to  the  default  slew  position,  pointing  to  the  nose 
of  the  aircraft. 
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COGNITIVE  SYSTEMS  ENGINEERING  (CSE) 
FRAMEWORK  FOR  EVALUATING  NEW  COCKPIT 
INTERFACES 


Kamilla  Run  Johannsdottir 
Jo- Anne  LeFevre,  Ph  D. 

Chris  M.  Herdman,  Ph  D. 

Centre  for  Applied  Cognitive  Research, 

Carleton  University,  Ottawa,  Canada 

With  an  increased  role  of  modeling  and  simulation  new  cockpit  technologies  can  be  thoroughly  evaluated  before 
being  implemented  in  an  aircraft.  This  process  of  evaluation,  however,  needs  to  be  guided  by  a  framework  based  on 
our  knowledge  of  the  strengths  and  limitations  of  the  human  operator.  Situation  awareness  (SA)  has  become  the 
dominant  construct  used  in  evaluating  new  technology.  However,  it  is  unlikely  that  a  single  construct  accurately 
captures  the  complexity  of  the  cognitive  processes  under  study.  Rather  than  using  a  single  construct  we  propose  a 
cognitive  systems  engineering  (CSE)  framework  for  guiding  the  modeling  and  simulation  process  in  evaluating  new 
cockpit  technologies.  The  CSE  framework  uses  converging  measures  of  three  central  constructs  (i.e.,  SA,  workload, 
and  task-relevant  performance)  to  operationalize  the  relevant  cognitive  processes  underlying  the  pilot-machine 
interaction.  It  is  argued  that  converging  measures  of  these  central  constructs  are  essential  for  providing  a 
comprehensive  perspective  on  the  impact  of  new  interfaces  on  the  pilot  and  the  crew. 


Introduction 

In  the  past,  the  implications  of  pilot-machine 
interactions  were  rarely  explored  thoroughly  when 
new  technology  was  added  to  existing  systems.  With 
the  development  of  affordable  simulation 
environments,  however,  the  feasibility  of  testing  new 
technologies  before  they  are  installed  in  aircraft  has 
increased  substantially.  Savings  in  terms  of  human 
costs  and  technology  retrofits  are  potentially 
enormous.  Furthermore,  there  have  been  recent 
advances  in  both  our  knowledge  of  human 
psychology  and  the  capabilities  of  simulation 
environments,  thus  supporting  a  greater  role  for 
modeling  and  simulation.  However,  it  is  important 
that  the  evaluation  process  of  modeling  and 
simulation  be  guided  by  a  proper  operationalization 
of  the  cognitive  processes  under  study. 

It  has  proven  difficult  to  properly  define  the  relevant 
cognitive  processes  for  empirical  evaluation  of  the 
pilot-machine  interface.  One  approach  to  this 
problem  is  for  researchers  to  focus  on  a  single 
construct  that  captures  a  substantial  and/or  relevant 
portion  of  the  pilots’  performance.  The  construct  of 
situation  awareness  (SA)  has  been  used  in  this  way, 
both  in  the  aviation  field  and  more  broadly  when 
researchers  have  explored  the  relations  between 
technology  and  human  performance.  SA  refers  to  the 
pilots’  conscious  comprehension  of  the  environment 
and  their  ability  to  project  future  scenarios  (Endsley, 


1995a,  2000).  Hence,  SA  is  most  commonly 
measured  with  various  questionnaires  whereby  pilots 
either  subjectively  evaluate  their  SA  or  the  reported 
knowledge  of  the  pilot  is  compared  to  the  actual  state 
of  the  system  and  the  environment  (Endsley,  1995b; 
Pew,  2000).  Less  commonly,  SA  is  also  evaluated  by 
measuring  task-relevant  performance,  which  is 
assumed  to  give  an  indirect  indication  of  pilots’ 
comprehension  of  the  situation  (Vidulich,  2000). 
Subjective  and  objective  measures  are  rarely  used 
together  to  evaluate  the  impact  of  new  technology  on 
the  pilot  or  the  crew. 

Measuring  SA  has  provided  some  important  insight 
into  the  impact  that  a  new  technology  may  have  on 
pilots’  comprehension  of  a  system  and  their  ability  to 
selectively  process  relevant  information.  However, 
the  pilot-machine  interaction  depends  on  a  complex, 
dynamic  model  of  the  system  and  the  situation  that 
includes  much  more  than  selective  attention  and 
comprehension.  For  example,  a  pilot’s  ability  to 
interact  with  a  system  is  also  highly  dependent  on 
perceptuomotor  coordination  and  action  planning.  It 
is  therefore  unlikely  that  a  single  construct  can 
sufficiently  capture  the  behavioural  variance 
produced  by  the  complex  pilot-machine  interaction. 
Furthermore,  a  single  construct  is  unable  to 
effectively  represent  cognitive  processes  that  underlie 
pilot-machine  interactions. 

It  is  important  that  the  constructs  that  are  chosen  to 
evaluate  the  pilot-machine  interaction  be 
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comprehensively  operationalized.  Using  a  single 
operation  seriously  limits  the  nature  and  the  meaning 
of  the  phenomena  under  study  (Gardner,  Hake  and 
Eriksen,  1956).  Due  to  the  complexity  of  the  pilot- 
machine  interaction  and  the  underlying  cognitive 
processes,  it  is  unlikely  that  a  single  measure  will 
provide  sufficient  information  to  allow  for  a  general 
conclusion  as  to  how  a  new  technology  affects  the 
pilot  or  the  crew.  Therefore,  in  order  to  properly 
operationalize  the  chosen  constructs,  multiple 
operations  or  measures  should  be  used. 

In  the  present  paper  we  propose  a  cognitive  systems 
engineering  (CSE)  framework  for  guiding  the 
modeling  and  simulation  process  in  evaluating  new 
cockpit  technologies.  The  CSE  framework  uses 
converging  measures  of  three  central  constructs  (i.e., 
SA,  workload,  and  task-relevant  performance)  to 
operationalize  the  relevant  cognitive  processes  under 
study.  We  argue  that  converging  measures  of  these 
central  constructs  are  essential  for  providing  a 
comprehensive  perspective  on  how  new  interfaces 
influence  pilots’  responses  in  the  cockpit. 

The  CSE  framework 

As  shown  in  Figure  1,  the  CSE  framework  includes  a 
theoretical  construct,  the  dynamic  mental  model,  and 
three  empirical  (i.e.,  measurable)  constructs:  situation 
awareness,  workload,  and  task-relevant  performance. 
Our  review  of  the  literature  indicated  that  these  three 
empirical  constructs  capture  a  significant  amount  of 
the  variance  in  the  pilot-machine  interface. 


The  proximal  goal  of  the  CSE  framework  is  to 
provide  a  context  within  which  to  interpret  the 


dependent  variables  that  are  assessed  in  a  modeling 
and  simulation  evaluation.  The  CSE  framework  is 
not  intended  to  represent  a  complete  model  either  of 
the  human  operator  or  of  the  situation,  although 
further  developments  of  the  framework  will  be 
designed  to  expand  the  theoretical  and  predictive 
power  of  the  model. 

The  central  theoretical  construct  in  the  CSE 
framework  is  the  dynamic  mental  model.  It  captures 
the  notion  that  the  human  operator  constantly  creates 
and  maintains  an  internal  representation  of  the 
ongoing  situation.  When  experimental  methods  are 
used  to  measure  performance  in  an  evaluation  of  new 
technology,  all  of  the  measurements  indirectly  index 
the  pilot’s  dynamic  mental  model.  The  three 
empirical  constructs  provide  a  comprehensive 
(although  not  exhaustive)  assessment  of  the  dynamic 
mental  model.  Workload  indexes  the  cognitive  effort 
required  by  the  pilot,  SA  captures  the  pilot’s 
perception  of  events  and  his  integration  of  those 
events  into  a  coherent  understanding  of  the  situation, 
and  task-relevant  performance  captures  the  behaviors 
associated  with  the  interactions  between  the  pilot  (or 
crew)  and  the  machine. 

SA  is  an  empirical  construct  in  the  CSE  framework 
that  can  be  defined  simply  as  “knowing  what  is  going 
on  around  you”  (Endsley,  2000,  p.  5).  The  term  was 
originally  used  to  capture  why  some  fighter  pilots 
were  more  successful  (and  therefore  lived  longer) 
than  others  (Spick,  1988).  SA  is  closely  tied  to 
knowing  how  to  distinguish  important  information  in 
the  environment  from  less  important  information 
(selective  attention),  as  well  as  the  ability  to  quickly 
comprehend  and  predict  the  importance  of  changes  to 
elements  in  the  environment  (Adams,  Tenney  and 
Pew,  1995;  Durso  and  Gronlund,  1999;  Endsley, 
1995a;  2000;  Sarter  and  Wood,  1995). 

Measuring  SA  provides  important  insight  into  how 
well  the  pilot  comprehends  a  system’s  functionality 
and  how  well  the  pilot  is  able  to  integrate  the 
information  presented  on  the  instrumentation  into  a 
coherent  picture  of  the  system  and  the  environment. 

The  second  empirical  construct  that  forms  the  core  of 
the  CSE  framework  is  workload.  Workload  is  an 
important  construct  in  aviation  and  other  complex 
cognitive  tasks  because  humans  are  limited  in  their 
ability  to  process  information  and  to  respond 
appropriately  (Hancock  and  Desmond,  2001).  A 
helicopter  pilot  on  a  search-and-rescue  mission  who 
is  flying  with  night  vision  goggles  near  the  ground 
during  a  rain  storm  is  likely  to  be  in  a  situation  of 
heavy  workload.  A  large  amount  of  rapidly  changing 
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information  has  to  be  monitored  and  the  pilot  must 
constantly  update  his  or  her  dynamic  mental  model  of 
the  environment.  In  contrast,  a  pilot  flying  a  routine 
transit  leg  in  good  weather  is  probably  in  a  low 
workload  situation. 

Workload  has  proven  to  be  a  very  useful  construct  for 
understanding  changes  in  pilots’  behaviour  under 
different  situational  demands  and  constraints  (Flach 
and  Kuperman,  2001;  Wickens,  2001).  Technology 
“improvements”  should  hypothetically  decrease 
workload,  but  in  practice,  a  technology  that  adds 
information  to  the  pilot’s  environment  and/or 
requires  the  pilot  to  perform  additional  tasks  may 
actually  increase  workload,  at  least  in  the  short  term 
(Vidulich,  2000).  Thus,  measuring  changes  in 
workload  as  a  function  of  technology  changes  is 
crucial  to  understanding  how  technology  influences 
pilots’  performance. 

The  third  empirical  construct  in  the  CSE  framework, 
task-relevant  performance,  refers  to  the  actions  of  the 
pilot  or  the  crew  (in  relation  to  mission  demands)  that 
are  potentially  affected  by  the  new  technology. 
Adding  a  new  technology  to  the  cockpit  can  affect 
pilots’  performance  in  various  ways.  For  example, 
the  addition  of  a  new  display  screen  to  the  F-18 
cockpit  that  has  enhanced  information  about 
approaching  threats  should  improve  pilots’  ability  to 
manoeuver  in  a  threatening  environment.  However, 
other  aspects  of  pilots’  behaviour  might  be  impaired 
or  affected  in  ways  that  decreases  their  SA  and/or 
interferes  with  how  they  interact  with  other  systems. 
For  example,  adding  the  new  display  screen  to  the  F- 
18  cockpit  should  result  in  the  pilot  spending  time 
looking  at  that  screen  and  interacting  with  it  in 
certain  ways.  Concomitantly,  the  pilot  may  spend  less 
time  using  other  sources  of  information,  or  may  use 
that  information  differently.  Thus,  defining  and 
measuring  task-relevant  performance  is  an  important 
aspect  of  understanding  the  impact  of  a  new 
technology  on  performance  in  the  cockpit. 

A  detailed  discussion  of  the  relations  among  SA, 
workload,  and  task-relevant  behaviour  is  beyond  the 
scope  of  the  present  paper.  In  brief,  however,  it  is 
clear  that  both  SA  and  workload  represent  outcomes 
that  may  not  be  directly  realized  in  overt  behaviour. 
Instead,  we  see  these  as  a  product  of  the  pilot’s 
creation  and  use  of  the  dynamic  mental  model. 
Therefore,  under  some  conditions,  we  would  expect 
to  find  a  high  correlation  between  SA  and  workload. 
If,  for  example,  a  decrease  in  workload  allows  pilots 
to  spend  more  time  scanning  the  environment  and 
detecting  dangerous  situations  more  quickly,  then  SA 
will  increase  as  workload  decreases.  In  our  view,  a 


complete  disconnection  between  workload  and  SA 
would  be  evidence  against  the  proposed  framework 
since  both  constructs  are  assumed  to  be  based  on  the 
pilots  creating  and  updating  their  mental  model. 

Good  SA  does  not  always  lead  to  good  performance 
and  high  workload  does  not  always  predict  poor 
performance.  For  example,  highly  trained  pilots  may 
function  very  well  under  high  workload  situations 
because  of  their  extensive  training  and  experience 
such  that  they  continue  to  function  effectively  despite 
increased  task  demands.  Nevertheless,  we  would 
predict  that  some  other  aspect  of  their  performance 
(such  as  SA)  might  decrease  under  heavy  cognitive 
demands.  The  CSE  framework  is  based  on  the 
assumption  that  the  three  empirical  constructs  will 
typically  be  related  such  that,  in  many  situations  good 
SA  (or  low  workload)  will  predict  good  performance. 
Hence,  if  task-relevant  performance  is 
operationalized  appropriately  as  a  specific  and  direct 
measurement  of  the  behaviour  that  is  likely  to  be 
affected  by  the  technology  change,  then  changes  in 
task-relevant  behaviour  should  be  correlated  with  SA. 
On  this  view,  if  task-relevant  behaviour  becomes 
worse  with  new  technology,  then  SA  must 
necessarily  decrease. 

In  summary,  according  to  the  CSE  framework,  the 
three  experimental  constructs  (SA,  workload,  task¬ 
relevant  performance)  are  second-order  reflections  of 
the  pilot’s  dynamic  mental  model  of  the  situation. 
Because  it  is  impossible  in  practice  to  directly 
measure  the  contents  of  the  dynamic  mental  model, 
defining  performance  relative  to  multiple  constructs 
that  access  the  mental  model  is  likely  to  provide  more 
useful  information  than  focusing  on  a  single 
construct. 

In  accord  with  the  CSE  framework,  we  propose  that 
the  evaluation  of  a  new  cockpit  technology  should 
include  at  least  one,  and  preferably  multiple  measures 
of  situation  awareness,  workload,  and  task-relevant 
performance  respectively.  Use  of  multiple  measures 
will  allow  for  a  richer  and  more  accurate  answer  to 
the  question  “how  does  the  new  technology  affect  the 
human-machine  interaction”? 

The  value  of  multiple  converging  measures  within  the 
CSE  framework 

The  pilot’s  internal  model  is  a  complex  theoretical 
construct  that  is  not  directly  measurable  (as  it  is  not 
possible  to  directly  measure  “memory”  or 
“thinking”),  and  therefore  the  only  measures  we  can 
use  are  behavioural  in  the  sense  that  the  pilot  must 
perform  some  action  that  is  then  assessed.  It  is  worth 
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emphasizing,  however,  that  unlike  other  cognitive 
constructs  such  as  attention  or  working  memory,  the 
dynamic  mental  model  represents  a  complex 
cognitive  mechanism  that  incorporates  various 
cognitive  processes,  such  as  selective  attention,  long¬ 
term  memory  and  perceptuomotor  coordination. 
Therefore  it  is  unlikely  that  a  single  construct  would 
capture  the  complexity  of  the  dynamic  mental  model. 

The  approach  taken  when  using  the  CSE  framework 
is  to  evaluate  the  pilot-machine  interaction  and  the 
underlying  dynamic  mental  model  by  using  the  three 
central  constructs  of  SA,  workload  and  performance. 
These  three  central  constructs  are  essentially 
behavioural  in  nature  and  separated  from  the 
theoretical  construct  of  the  dynamic  mental  model  to 
avoid  the  risk  of  conceptual  confusion  where  the 
defined  measured  behaviour  is  the  same  as  the 
underlying  cognitive  mechanism  (see  for  example, 
Flach,  1995).  However,  the  three  empirical  constructs 
of  SA,  workload  and  performance  must  also  be 
comprehensively  represented  through  the  use  of 
multiple  operations.  This  is  because  it  is  unlikely  that 
any  single  measure  can  provide  sufficient  information 
to  allow  for  general  conclusions  about  the  impact  of  a 
new  technology  on  the  user.  Hence,  in  addition  to 
assessing  the  three  central  constructs,  we  propose  that 
researchers  collect  multiple  measures  for  each 
construct. 

An  important  distinction  that  is  often  overlooked  in 
aviation  research  is  between  subjective  and  objective 
measures  of  constructs  such  as  SA  or  workload.  A 
subjective  assessment,  for  example,  of  situation 
awareness  requires  that  the  pilot  makes  a  judgment  or 
provides  an  evaluation.  Pilots  might  be  asked  to 
report  on  where  they  were  looking  during  a 
manoeuver  and  what  information  they  noticed.  A 
researcher  could  objectively  measure  SA  by 
measuring  the  pilots’  head  position  as  an  index  of 
where  the  pilot  is  attending.  It  is  critical  to 
distinguish  between  objective  and  subjective 
measures  because  an  individual’s  perception  of  their 
behaviour  or  memory  for  a  situation  can  be  wrong. 
Subjective  measures,  for  example,  might  be  better 
characterized  as  perceived  workload  or  perceived  SA. 

Subjective  measures  allow  us  to  evaluate  pilots’ 
degree  of  comfort  and  acceptance  of  the  new 
technology  and  as  such  can  be  extremely  informative. 
For  example,  it  is  possible  that  adding  a  new  cockpit 
system  does  not  significantly  affect  pilots’ 
performance.  However,  their  level  of  acceptance  and 
discomfort  may  increase  significantly  and  later  cause 
performance  decrement.  This  is  particularly  true  for 
high  stress  situations  where  workload  is  suddenly 


increased  (Andre,  2001).  Similarly,  pilots  may 
subjectively  prefer  a  new  system  to  existing  system 
whereas  their  performance  is  being  impaired  by  the 
new  technology.  Therefore,  using  both  subjective  and 
objective  measures  will  provide  a  more 
comprehensive  evaluation  of  the  impact  of  new 
technology  on  the  pilot-machine  interaction. 

In  a  recent  study,  we  demonstrated  the  benefits  of 
using  multiple  measures  to  evaluate  the 
implementation  of  a  direct  voice  input  (DVI)  system 
for  controlling  the  on-board  computer  in  the  CHI 46 
Griffon  helicopter  (Herdman  et  al.,  2001;  Lessard  et 
al.,  2001).  In  this  study,  heads-up  time  was  identified 
as  a  key  to  enhancing  SA.  Increased  heads-ups  time 
should  improve  pilots’  ability  to  detect  and  respond 
to  events  in  the  external  scene.  Heads-up  time  was 
measured  by  tracking  the  pilots’  head  position 
throughout  simulated  reconnaissance  missions. 
Heads-up  time  in  the  DVI  condition  was  compared  to 
a  standard  manual  input  condition  that  was 
configured  based  on  how  the  Griffon  crew  currently 
enters  commands  into  the  CDU.  As  predicted,  heads- 
up  time  increased  with  DVI  relative  to  the  manual 
condition  (by  an  impressive  42%),  indicating  that  the 
technology  change  had  at  least  one  of  the  expected 
and  desirable  outcomes  on  pilots’  behaviour. 

However,  it  was  recognized  that  introducing  the  DVI 
system  to  the  Griffon  cockpit  has  a  variety  of  other 
potential  effects.  First,  DVI  had  the  potential  to 
change  the  crew  interactions  in  that  the  flying  pilot 
now  had  the  opportunity  to  control  the  on-board 
computer  (i.e.,  the  CDU),  whereas  in  the  manual 
input  situation  only  the  non-flying  pilot  can  enter 
commands  on  the  CDU.  So  the  workload  or 
performance  of  the  flying  pilot  might  also  be  affected 
by  the  new  technology.  Second,  if  looking  at  the 
CDU  to  manually  enter  commands  was  a  workload¬ 
intensive  activity  for  the  non-flying  pilot,  then  DVI 
might  decrease  his  or  her  overall  workload.  Other 
aspects  of  the  pilots’  task  performance,  however, 
were  unlikely  to  be  affected  in  the  types  of  missions 
that  were  flown. 

Herdman  et  al.  (2001)  included  objective  measures  to 
assess  the  workload  demands  of  the  DVI  versus 
manual  input  systems.  Objective  workload  was 
measured  using  detection  of  auditory  and  visual 
stimuli  (i.e.,  targets)  by  both  pilots.  The  targets 
(auditory  tones  or  visual  flashes  in  the  external  scene) 
were  presented  randomly  in  the  course  of  the 
simulated  missions.  Pilots  were  instructed  to  respond 
as  quickly  as  possible  when  they  detected  a  target  by 
pressing  a  key.  Target  detection  as  an  index  of 
workload  has  been  used  extensively  and  thus  has 
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both  empirical  and  theoretical  support.  Essentially, 
the  speed  and  accuracy  with  which  pilots  respond  to 
the  auditory  and/or  visual  targets  is  used  as  a  measure 
of  their  available  attentional  capacity. 

Herdman  et  al.  (2001)  found  that  the  workload  of  the 
non-flying  pilots  was  less  in  the  DVI  condition  than 
in  the  manual  input  condition.  It  was  concluded, 
therefore  that  for  the  non-flying  pilots,  the  DVI 
system  should  improve  SA  and  lower  workload. 
Interestingly,  it  was  found  that  the  workload  of  the 
flying  pilots  increased  significantly  in  the  DVI 
condition.  This  increase  in  workload  occurred  even 
though  the  flying  pilots  used  the  DVI  capability 
infrequently  (i.e.,  less  than  1  minute  DVI  time  per 
each  20  minute  mission).  Subjective  measures  of  the 
flying  pilot’s  workload  and  SA  did  not  differ  across 
the  DVI  versus  manual  input  condition,  however. 
These  results  support  our  contention  that  a  broad 
assessment  of  multiple  constructs  is  necessary  to 
achieve  a  comprehensive  understanding  of  the  impact 
of  a  new  technology. 

The  example  from  the  DVI  study  emphasizes  the 
importance  of  using  converging  measures  to  properly 
evaluate  the  impact  of  new  technology  on  the  pilot 
and  the  crew.  Research  using  the  CSE  framework 
and  the  principles  of  broad  assessment  will  test  the 
usefulness  of  this  approach.  These  techniques  are  not 
complicated  to  apply.  For  example,  to  assess  the  use 
of  a  new  altimeter,  altitude  maintenance  could  be 
used  as  an  index  of  the  pilot’s  adherence  to  the  flight 
plan  (task-relevant  behaviour),  the  simulation  can  be 
frozen  and  pilots  could  be  asked  to  report  altitude 
information  (situation  awareness).  Their  workload 
could  be  measured  (using  target  detection)  with  the 
new  and  old  instillments. 

Systematic  assessment  of  all  three  constructs  with 
both  subjective  and  objective  indices  would  allow  for 
a  comprehensive  picture  of  how  the  new  technology 
influences  the  pilot-machine  interaction.  By 
including  multiple  measures  of  the  three  behavioural 
constructs  a  more  complete  picture  can  be  inferred 
about  the  underlying  cognitive  processes. 

In  summary,  the  CSE  framework  encourages 
researchers  to  develop  measures  that  assess  pilots’ 
behaviour  from  multiple  perspectives.  The 
framework  brings  different  measures  and  different 
definitions  of  the  pilot-machine  interaction  together 
in  a  single  framework  that  will  allow  us  to  more 
thoroughly  evaluate  the  implications  of  new  cockpit 
technology  for  the  pilot.  The  central  assumption  is 
that  using  converging  measures  of  these  three  central 
constructs  (i.e.,  SA,  workload,  and  task-relevant 


performance),  will  provide  a  comprehensive 
perspective  on  how  new  interfaces  influence  pilots’ 
responses  in  the  cockpit. 

Conclusions 

The  increasing  complexity  of  the  modern  cockpit 
calls  for  the  development  of  tools  and  methods  that 
allows  us  to  evaluate  the  impact  of  new  cockpit 
technology  on  the  pilot  and  the  crew.  Flowever,  such 
evaluation  tools  must  be  guided  by  a  proper 
operationalization  of  the  relevant  cognitive  processes. 
The  present  paper  proposes  a  cognitive  systems 
engineering  (CSE)  framework  that  uses  converging 
measures  of  central  constructs  (SA,  workload,  and 
task  relevant  performance)  to  evaluate  how  the  pilot- 
machine  interaction  is  affected  by  new  technology. 

A  central  tenet  of  the  CSE  framework  is  that  the 
pilot-machine  interaction  in  the  cockpit  is  too 
complex  for  a  single  construct  to  provide  sufficient 
information  to  evaluate  the  impact  of  an  interface  on 
pilots’  overall  behaviour.  By  using  converging 
measures  of  the  three  central  constructs  it  is  more 
likely  that  we  are  capturing  the  relevant  cognitive 
processes  we  want  to  evaluate.  Furthermore,  by 
distinguishing  among  SA,  workload,  and  task¬ 
relevant  performance,  and  the  underlying  mental 
representation  (the  dynamic  mental  model), 
researchers  can  more  clearly  operationalize  the 
concepts  for  empirical  purposes. 

In  particular,  the  CSE  framework  proposes  the  use  of 
both  subjective  and  objective  measures  for  each  of 
the  three  empirical  constructs  of  SA,  workload  and 
performance.  This  is  because  subjective  and  objective 
measures  of  the  same  construct  can  produce  different 
outcomes.  For  example,  individual’s  perception  of 
their  behaviour  or  memory  for  a  situation  can  diverge 
considerably  from  their  actual  performance.  It  is 
argued  within  the  CSE  framework  that  using  multiple 
operations  of  both  subjective  and  objective  measures 
will  provide  a  more  comprehensive 
operationalization  of  SA,  workload  and  performance 
and  as  such  will  provide  a  more  complete  evaluation 
of  how  a  new  technology  affects  the  pilot  and  the 
crew. 

The  CSE  framework  is  expected  to  provide  important 
support  to  the  modelling  and  simulation  process  in 
evaluating  new  cockpit  technology.  Research  that  is 
conducted  within  the  CSE  framework  should  allow 
for  comprehensive  and  valid  assessment  of  human 
factors  aspects  of  new  aviation  technology. 

Systematic  application  of  the  framework  in  the 
evaluation  of  new  technology  for  the  cockpit  will 
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allow  researchers  to  evaluate  the  results  of 
assessments  produced  by  different  labs  under 
different  conditions  to  be  compared  more  easily. 
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Summary 


Military  aircraft  provide  a  forum  for  the  development  of  new  aviation  technology  that  has 
potentially  wide  application.  For  example,  heads-up  displays,  a  technology  that  is  common  in 
military  contexts,  are  just  becoming  widespread  in  commercial  aircraft.  In  the  past,  the 
implications  of  human-machine  interactions  were  rarely  explored  thoroughly  when  new 
technology  was  added  to  existing  systems.  With  the  development  of  affordable  simulation 
environments,  however,  the  feasibility  of  testing  new  technologies  before  they  are  installed  in 
aircraft  has  increased  substantially.  Savings  in  terms  of  human  costs  and  technology  retrofits  are 
potentially  enormous.  Furthermore,  there  have  been  recent  advances  in  both  our  knowledge  of 
human  psychology  and  the  capabilities  of  simulation  environments,  supporting  a  greater  role  for 
modeling  and  simulation  (i.e.,  M&S).  In  this  report,  we  propose  a  cognitive  systems  engineering 
(CSE)  framework  to  guide  the  evaluation  component  of  the  M&S  process.  The  goal  of  the  CSE 
framework  is  to  provide  general  guidelines  for  evaluating  new  technologies  from  the  perspective 
of  how  these  will  affect  the  human-machine  interface.  This  initial  report  consists  of  three  parts. 
Section  I  is  an  overview  of  the  proposed  CSE  framework.  Section  II  is  a  more  detailed  literature 
review  upon  which  the  framework  was  based.  Section  III  is  a  comprehensive  bibliography. 
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Section  I:  Proposed  Cognitive  Systems  Engineering  Framework 

I. A.  Overview  of  the  Proposed  Framework 

The  proposed  framework  for  evaluating  new  technologies  in  the  M&S  process  is  based 
on  the  application  of  a  human  factors  approach  to  understanding  cockpit  design.  In  the 
Cognitive  Systems  Engineering  (CSE)  framework,  we  assume  that  the  limitations  of  the  human, 
rather  than  of  the  technology,  must  guide  the  development  of  new  aviation  systems.  Pilots  are 
extremely  highly  skilled  operators  of  extremely  complex  machines.  Improvements  in  technology 
must  be  designed  around  an  understanding  of  the  strengths  and  limitations  of  the  human 
operator.  Thus,  the  proposed  CSE  framework  incorporates  understanding  of  human  cognition  in 
a  working  blueprint  for  the  design  of  evaluation  experiments  to  support  modeling  and  simulation 
in  acquisition.  As  shown  in  Figure  1.1,  the  framework  includes  a  theoretical  construct,  the 
dynamic  mental  model,  and  three  empirical  (i.e.,  measurable)  constructs;  situation  awareness, 
workload,  and  task-relevant  perfonnance.  Our  review  of  the  literature  indicated  that  these  three 
empirical  constructs  capture  a  large  amount  of  the  variance  in  the  human-machine  interface.  The 
proximal  goal  of  the  CSE  framework  is  to  provide  a  context  within  which  to  interpret  the 
dependent  variables  that  are  assessed  in  an  M&S  evaluation  (i.e.,  how  to  assess  the  benefits  of  a 
new  technology  in  a  complex  environment  such  as  the  Griffon  CH146  helicopter).  It  is  not 
intended  to  represent  a  complete  model  either  of  the  human  operator  or  of  the  situation,  although 
further  developments  of  the  framework  will  be  designed  to  expand  the  theoretical  and  predictive 
power  of  the  model. 

The  central  theoretical  construct  in  the  CSE  framework  is  the  dynamic  mental  model.  It 
captures  the  notion  that  the  human  operator  constantly  creates  and  maintains  an  internal 
representation  of  the  ongoing  situation.  When  experimental  methods  are  used  to  measure 
performance  in  an  M&S  evaluation,  all  of  the  measurements  are  inferences  about  the  operator’s 
dynamic  mental  model.  The  three  empirical  constructs,  situation  awareness,  workload,  and  task¬ 
relevant  performance,  are  assumed  to  provide  a  comprehensive  (although  not  exhaustive) 
assessment  of  the  dynamic  mental  model.  Situation  awareness,  defined  simply  as  “knowing 
what  is  going  on  around  you”  (Endsley,  2000,  p.  5)  is  a  construct  that  was  proposed  originally  to 
capture  why  some  fighter  pilots  were  more  successful  (and  therefore  lived  longer)  than  others 
(Spick,  1988).  As  discussed  by  Endsley  (2000),  SA  is  closely  tied  to  knowing  how  to  distinguish 
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important  infonnation  in  the  environment  from  less  important  infonnation  (selective  attention), 
as  well  as  the  ability  to  quickly  comprehend  the  importance  of  changes  to  elements  in  the 
environment. 

The  second  empirical  construct  that  forms  the  core  of  the  CSE  framework  is  workload. 
Workload  is  a  familiar  construct  in  aviation  and  has  been  studied  more  thoroughly  than  SA. 
Essentially,  workload  refers  to  the  fact  that  humans  are  limited  in  their  ability  to  process 
information  and  to  respond  appropriately.  A  helicopter  pilot  on  a  SAR  mission  who  is  flying 
with  night  vision  goggles  near  the  ground  during  a  rain  storm  is  likely  to  be  in  a  situation  of 
heavy  workload.  A  large  amount  of  information  that  changes  rapidly  has  to  be  monitored  and 
the  pilot  must  constantly  be  updating  his  or  her  dynamic  mental  model  of  the  environment.  In 
contrast,  a  fighter  pilot  flying  in  good  weather  on  a  routine  recce  is  probably  in  a  low  workload 
situation.  Workload  has  proven  to  be  a  very  useful  construct  for  understanding  changes  in  pilots’ 
behaviour  under  different  situational  demands  and  constraints.  Technology  “improvements” 
should  hypothetically  decrease  workload,  but  in  practice,  a  technology  that  adds  infonnation  to 
the  pilot’s  environment  and/or  requires  the  pilot  to  perform  additional  tasks  is  more  likely  to 
increase  workload.  Thus,  measuring  changes  in  workload  as  a  function  of  technology  changes  is 
crucial  to  understanding  how  that  technology  influences  pilots’  performance. 

The  third  empirical  construct  in  the  framework,  task-relevant  performance,  refers  to  the 
actions  of  the  pilot  (in  relation  to  mission  demands)  that  are  potentially  affected  by  the  new 
technology.  In  essence,  a  new  technology  is  expected  to  change  some  aspect  of  what  the  pilot 
knows  and  that  knowledge  will  be  reflected  in  his  or  her  behaviour.  Many  other  aspects  of 
pilots’  actions  or  behaviour  might  not  change,  however.  The  task-relevant  performance  that  is 
relevant  to  any  particular  technological  change  will  depend  on  what  that  technology  was 
expected  to  influence.  For  example,  the  addition  of  a  new  display  screen  to  the  F-18  cockpit  that 
has  enhanced  information  about  approaching  threats  should  result  in  the  pilot  spending  time 
looking  at  that  screen  and  interacting  with  it  in  certain  ways.  Concomitantly,  the  F-18  pilot  may 
spend  less  time  using  other  sources  of  information,  or  may  use  that  information  differently. 

Thus,  defining  and  measuring  task-relevant  perfonnance  is  an  important  aspect  of  understanding 
the  impact  of  a  new  technology  on  performance  in  the  cockpit. 
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FIGURE  1.1  -  CSE  FRAMEWORK 
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In  the  CSE  framework,  the  three  experimental  constructs  (SA,  workload,  task-relevant 
performance)  are  second-order  reflections  of  the  pilot’s  dynamic  mental  model  of  the  situation. 
Because  it  is  impossible  in  practice  to  directly  measure  the  contents  of  a  human  brain,  use  of 
multiple  measures  of  the  dynamic  mental  model  is  likely  to  provide  more  useful  information  than 
focusing  on  a  single  construct  such  as  situation  awareness.  Furthermore,  information  acquired 
through  the  research  process  is  limited  by  the  types  of  measures  chosen  and  by  the  underlying 
theoretical  assumptions  that  guided  the  choice  of  those  measures.  In  Section  II,  we  present  a 
detailed  overview  of  the  strengths  and  weaknesses  of  the  constructs  of  situation  awareness  and 
workload.  The  bottom-line  conclusion  is  that  any  single  measure  is  unlikely  to  provide  sufficient 
information  to  allow  for  general  conclusions  about  the  impact  of  a  new  technology  on  the  crew. 

Because  it  is  impossible  to  directly  measure  the  dynamic  internal  model  (as  it  is  not 
possible  to  directly  measure  “memory”  or  “thinking”),  all  measures  we  can  use  are  behavioural 
in  the  sense  that  the  pilot  or  crew  must  perfonn  some  action  that  is  then  assessed.  For  example,  a 
subjective  assessment  of  situation  awareness  requires  that  the  pilot  make  a  judgment  or  provide 
an  evaluation.  Head  position  could  be  used  as  an  index  of  where  the  pilot  is  attending.  Altitude 
maintenance  could  be  used  as  an  index  of  the  pilot’s  adherence  to  the  flight  plan.  In  the  case  of 
SA,  the  simulation  can  be  frozen  and  then  pilots  can  be  asked  to  report  instrument  information 
(such  as  altitude),  or  to  make  a  prediction  about  the  trajectory  of  enemy  planes  (i.e.,  bogies). 
Their  answers  to  such  queries  are  assumed  to  reflect  their  interrogation  of  their  dynamic  mental 
model  (i.e.,  memory  for  the  current  location  of  bogies  and  knowledge  about  what  those  bogies 
are  likely  to  do).  In  general,  the  researcher  assumes  that  the  behavioural  responses  or  actions  can 
be  used  as  an  index  of  the  pilot’s  knowledge  or  mental  processes. 

Because  of  the  complexity  and  multi-dimensional  nature  of  the  dynamic  mental  model, 
any  single  construct  or  any  single  measure  of  a  construct  is  unlikely  to  capture  sufficient 
information  about  the  impact  of  a  new  technology.  Instead,  we  propose  that  the  CSE  evaluation 
should  include  at  least  one,  and  preferably  multiple  measures  of  each  of  situation  awareness, 
workload,  and  task-relevant  performance.  Use  of  multiple  measures  will  allow  for  a  richer  and 
more  accurate  answer  to  the  question  “how  does  the  new  technology  affect  the  human-machine 
interaction”?  Below,  we  provide  a  detailed  example  of  a  previous  M&S  evaluation  carried  out 
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by  the  CACR  (in  conjunction  with  CMC  Electronics)  that  illustrates  the  importance  of  having 
multiple  constructs  and  measures.  Although  this  evaluation  was  conducted  prior  to  the 
development  of  the  CSE  framework,  it  has  characteristics  that  are  consistent  with  the  CSE 
approach  that  we  are  recommending. 

I.B  -  Example:  Direct  Voice  Input  for  the  CH146  Griffon  Helicopter 

In  designing  and  selecting  measures  (whether  of  performance,  SA,  or  workload),  it  is 
important  that  these  be  constrained  both  by  the  specific  situation  (e.g.,  the  CH146  environment) 
and  the  nature  of  the  technological  upgrade.  Thus,  the  first  step  in  an  M&S  evaluation  is  to  do  a 
rational  analysis  of  how  the  technology  is  expected  to  influence  pilot’s  behaviour.  In  the 
evaluation  of  direct  voice  input  (DVI)  for  the  Griffon,  heads-up  time  was  identified  as  a  key 
variable  that  should  be  affected  when  pilots’  were  given  DVI  capabilities.  Furthermore,  heads- 
up  time  is  an  ecologically  valid  measure,  because  pilots  are  instructed  and  trained  to  spend  as 
much  time  as  possible  looking  outside  of  the  aircraft.  The  DVI  interface  that  was  designed  for 
the  Griffon  allowed  the  non-flying  pilot  to  use  voice  commands  for  certain  common  actions  that 
normally  would  be  entered  on  the  onboard  computer  (the  CDU)  that  is  located  between  the  seats 
in  the  cockpit.  For  example,  pilots  could  change  the  radio  frequency  by  saying  “SET  RADIO  1 
TO  121.5”  instead  of  typing  a  series  of  commands  on  the  CDU.  Using  a  voice  command  meant 
that  the  non-flying  pilot  could  potentially  keep  his  or  her  eyes  on  the  outside  world.  Because 
pilots  are  instructed  to  look  outside  the  cockpit  as  much  as  possible,  any  increase  in  heads-up 
time  would  be  a  reasonable  and  logical  outcome  of  the  change  from  manual  input  to  DVI. 
Heads-up  time  was  measured  in  this  evaluation  by  tracking  the  pilots’  head  position  throughout 
the  simulated  recce  missions,  and  then  comparing  the  total  heads-up  time  in  the  DVI  condition  to 
that  in  the  manual  input  position.  As  predicted,  heads-up  time  increased  with  DVI  relative  to  the 
manual  condition  (by  an  impressive  42%),  indicating  that  the  technology  change  had  at  least  one 
of  the  expected  and  desirable  outcomes  on  pilots’  behaviour. 

Heads-up  time  is  not  the  whole  story,  however.  Although  the  CSE  framework  shown  in 
Figure  1  had  not  been  developed  when  the  DVI  study  was  planned,  the  researchers  who 
conducted  the  evaluation  recognized  that  multiple  measures  of  behaviour  would  provide  a  more 
complete  picture  of  the  impact  of  DVI  on  the  human-machine  interaction.  In  particular, 
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introduction  of  DVI  to  the  Griffon  cockpit  had  a  variety  of  other  potential  effects.  First,  DVI  had 
the  potential  to  change  the  CRM  in  that  the  flying  pilot  now  had  the  opportunity  to  interact  with 
the  CDU,  whereas  in  the  manual  input  situation  only  the  non-flying  pilot  can  enter  commands  on 
the  CDU.  So  the  workload  or  performance  of  the  flying  pilot  might  also  be  affected  by  the  new 
technology.  Second,  if  having  to  look  at  the  CDU  to  enter  commands  was  a  workload-intensive 
activity  for  the  nonflying  pilot,  then  DVI  might  decrease  his  or  her  overall  workload.  Other 
aspects  of  the  pilots’  task  performance,  however,  were  unlikely  to  be  affected  in  the  types  of 
missions  that  were  flown. 

Workload  was  measured  in  this  experiment  using  detection  of  auditory  and  visual  stimuli 
(i.e.,  targets)  by  both  pilots.  The  targets  (auditory  tones  or  visual  flashes  in  the  external  scene) 
were  presented  randomly  in  the  course  of  the  simulated  missions.  Pilots  were  instructed  to 
respond  as  quickly  as  possible  when  they  detected  a  target  by  pressing  a  key.  Target  detection  as 
an  index  of  workload  has  been  used  extensively  and  thus  has  both  empirical  and  theoretical 
support.  Essentially,  the  speed  with  which  the  pilot  responds  to  the  target  can  be  used  as  a 
measure  of  his  or  her  available  mental  capacity.  In  the  DVI  experiment,  the  workload  of  the 
non-flying  pilot  was  less  in  the  DVI  condition  that  in  the  manual  condition,  suggesting  that  the 
DVI  facility  was  likely  to  result  in  improved  pilot  performance.  Importantly,  however,  the 
workload  of  the  flying  pilot  increased  significantly  in  the  DVI  condition.  Even  though  the  flying 
pilots  infrequently  used  the  DVI  capability,  the  incremental  change  to  their  responsibilities  was 
evident  in  the  workload  measure.  In  accord  with  the  CSE  framework,  DVI  had  a  significant 
impact  on  the  flying  pilot’s  dynamic  mental  model.  This  unexpected  finding  speaks  to  the 
complexity  of  changing  the  CRM  as  a  function  of  the  technological  upgrades  and  to  the 
importance  of  measuring  multiple  aspects  of  the  overall  pilot  activities. 

Situation  awareness  was  not  directly  measured  in  the  DVI  experiment.  Instead,  it  was 
inferred  that  SA  would  be  better  as  heads-up  time  increased  and  workload  decreased  (Vidulich, 
2000).  Because  inclusion  of  DVI  capability  resulted  in  a  substantial  increase  in  heads-up  time 
and  a  decrease  in  workload  for  the  nonflying  pilot,  it  was  concluded  that  the  SA  of  nonflying 
pilots  was  likely  to  improve  given  DVI  capabilities.  In  contrast,  the  SA  of  the  flying  pilot  may 
be  worse  in  DVI  conditions,  at  least  when  they  are  allowed  to  use  the  DVI  for  commands  that 
normally  would  be  handled  by  the  nonflying  pilot.  In  summary,  even  though  the  DVI  was 
conducted  before  our  fonnalization  of  the  CSE  framework,  it  provides  a  concrete  example  of 
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how  multiple  measures  and  converging  evidence  is  necessary  in  the  CSE  process.  Actual 
assessment  of  situation  awareness  would  have  provided  an  even  more  comprehensive  picture  of 
how  DVI  influenced  cockpit  activities. 

I.C  -  Implications  of  the  CSE  Framework  for  Cockpit  Research 

Our  review  of  the  literature  on  cockpit  research  shows  that  workload,  SA,  and  task-relevant 
performance  capture  a  substantial  amount  of  variability  in  the  human-machine  interaction. 
Workload  refers  to  the  cognitive  effort  required  by  the  pilot,  SA  is  the  pilot’s  perception  of 
events  and  integration  of  those  events  into  a  coherent  understanding  of  the  situation,  and  task¬ 
relevant  perfonnance  captures  the  behaviours  associated  with  the  interactions  between  the 
human  and  the  machine.  Direct  evaluation  of  performance  provides  important  infonnation 
about  the  pilot-system  interface  at  the  level  of  perceptual-motor  coordination  and  thus  allows  for 
the  evaluation  of  direct  interaction  with  the  system  through  systems  control.  Both  SA  and 
workload  represent  an  aspect  of  the  pilot-system  interaction  that  does  not  directly  reflect 
performance.  Therefore,  under  some  conditions,  we  should  not  expect  a  high  correlation  between 
SA  or  workload  and  the  overall  pilot-system  interaction.  Good  SA  does  not  always  lead  to  good 
performance  and  high  workload  does  not  always  predict  poor  performance.  Nevertheless,  these 
constructs  are  related  and  in  many  situations,  good  SA  will  predict  good  performance.  More 
generally,  the  CSE  framework  emphasizes  the  importance  of  multiple  interacting  aspects  of  pilot 
behaviour. 

Furthermore,  the  CSE  framework  makes  an  important  distinction  between  the  concepts  that 
are  being  measured  as  part  of  the  pilots’  behaviour  and  concepts  that  are  being  measured  as  the 
underlying  cognitive  mechanism.  One  of  the  major  problems  in  properly  defining  SA  is  a  lack  of 
distinction  between  SA  as  the  measured  construct  and  SA  as  the  pilot’s  internal  representation  of 
the  world  (e.g.,  Endsley,  2000).  SA  originated  as  a  description  of  behaviour  (i.e.,  that  some 
pilots  are  much  more  skilled  than  other  pilots)  but  has  often  been  used  in  the  literature  as  the 
explanation  for  good  performance  (e.g.,  because  the  pilot  has  good  SA,  she  performed  well). 

This  confusion  between  SA  as  an  explanation  and  SA  as  a  description  needs  to  be  avoided  so 
that  SA  does  not  become  circular  and  therefore  an  empty  concept  (Flach,  1995).  . 

A  central  tenet  of  the  CSE  framework,  therefore,  is  that  the  human-machine  interaction  in 
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the  cockpit  is  too  complex  for  a  single  concept  to  provide  sufficient  information  to  evaluate  the 
impact  of  an  interface  on  pilots’  overall  behaviour.  By  distinguishing  among  SA,  workload,  and 
performance  and  the  underlying  mental  representation  (the  dynamic  mental  model),  researchers 
can  more  clearly  operationalize  the  concepts  for  empirical  purposes.  Research  that  is  conducted 
within  the  CSE  framework  should  allow  for  comprehensive  and  valid  assessment  of  human 
factors  aspects  of  new  aviation  technology. 
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Section  II:  Definitions  and  Literature  Review 

Reviews  of  situation  awareness,  workload,  and  related  issues  in  cockpit  research  were 
done  with  various  databases,  including  PsycINFO,  Social  Science  Citation  Index  (SSCI), 
CISTISource  and  Web  of  Science.  Pertinent  references  from  other  sources  (e.g.,  conference 
proceedings  of  the  Aviation  Psychology  conference)  were  also  included.  The  edited 
bibliography  of  articles  gathered  via  these  searches  is  included  at  the  end  of  this  initial  report.  A 
reference  list  that  includes  those  cited  in  the  body  of  this  report  is  also  included.  Our  goal  with 
the  review  portion  of  this  report  is  to  provide  a  critical  overview  of  these  psychological 
constructs,  and  to  give  recommendations  that  are  pertinent  to  modeling  and  simulation  activities 
relevant  to  the  CHI 46.  A  third  section  (to  be  added  in  the  final  version  of  the  report)  will 
include  descriptions  of  the  results  of  Experiments  1  through  3,  where  the  CSE  framework  was 
used  to  develop  and  implement  the  human  factors  component  of  the  M&S  process. 

II. A.  -  Situation  Awareness 

II.A.1.  -  Defining  Situation  Awareness 

Situation  awareness,  defined  simply  as  “knowing  what  is  going  on  around  you”  (Endsley, 
2000,  p.  5)  is  a  construct  that  was  proposed  originally  to  capture  why  some  fighter  pilots  were 
more  successful  (and  therefore  lived  longer)  than  others  (Spick,  1988).  As  discussed  by  Endsley 
(2000),  SA  is  closely  tied  to  knowing  how  to  distinguish  important  information  in  the 
environment  from  less  important  information  and  to  the  ability  to  quickly  comprehend  the 
importance  of  changes  to  elements  in  the  environment  (Adams,  Tenney,  &  Pew,  1995;  Durso  & 
Gronlund,  1999;  Endsley,  1995;  2000;  Sarter  &  Woods,  1995).  Spick  (1988)  suggests  that  an 
important  aspect  of  what  made  the  aces  better  than  the  average  pilot  was  their  superior  judgment 
about  when  to  either  engage  in  a  fight  or  disengage  in  one  that  was  not  going  well.  Hence,  SA 
represents  pilots’  ability  to  keep  track  of  objects  such  as  enemy  aircraft  in  the  environment  and  to 
estimate  the  probability  of  success  in  a  given  situation. 

Situation  awareness  is  appealing  because  it  seems  intuitively  obvious  that  successful 
completion  of  missions  requires  that  pilots  are  aware  of  the  status  of  the  aircraft  and  the  various 
elements  of  the  situation.  It  has  proven  difficult,  however,  to  define  what  SA  is  for  empirical 


12 


CSE  Framework 


purposes.  Researchers  offer  different  perspectives  on  what  constitutes  SA.  This  lack  of  a  clear 
definition  has  made  it  difficult  to  establish  a  coherent  framework  for  conducting  SA  research. 
Furthermore,  researchers  disagree  as  to  whether  or  not  SA  is  not  a  unitary  phenomenon  (Pew, 
2000;  Sarter  &  Woods,  1991).  Researchers  face  a  variety  of  difficulties  in  defining  SA.  First, 
SA  encompasses  a  wide  variety  of  factors  such  as  keeping  track  of  objects  in  the  environment, 
attending  to  relevant  infonnation,  and  pilot’s  knowledge  of  the  mission.  Second,  the  elements  of 
the  situation  that  are  relevant  vary  considerably  depending  on  the  nature  of  the  mission  and  the 
aircraft.  For  example,  although  altitude  tracking  is  critical  to  a  helicopter  pilot  attempting  to 
maintain  a  hover  over  the  ocean,  altitude  tracking  is  not  critical  to  a  fighter  pilot  who  is  engaged 
with  an  enemy  aircraft.  These  complexities  have  made  it  difficult  to  produce  a  single,  definitive 
description  of  SA. 

One  commonality  across  different  descriptions  of  SA  is  an  emphasis  on  the  underlying 
cognitive  mechanisms  that  determine  SA  and  that  are  common  to  different  situations  (Adams  et 
al.,  1995;  Endsley,  2000).  Endsley  (1988)  defined  SA  as:  “...the  perception  of  the  elements  in 
the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the 
projection  of  their  status  in  the  future”  (1988,  p.  97).  Endsley  included  three  levels  within  the 
construct  of  SA:  perception,  comprehension  and  projection  (Endsley,  1995,2000).  Endsley’s 
model  will  be  considered  in  some  detail  because  it  is  the  most  widely  cited  description  of  the 
construct  of  SA. 

Level  1  SA  is  the  operator’s  perception  of  the  current  status  of  the  environment.  For 
instance,  in  a  cockpit  environment,  Level  1  SA  includes  the  perception  of  the  various  indicators 
on  the  display  such  as  the  altitude  indicator,  heading,  and  relative  position.  Level  1  also  includes 
the  perception  of  objects  in  the  environment  such  as  mountains,  trees,  other  aircraft,  and  warning 
lights.  Level  2  SA  refers  to  how  the  situation  is  understood  (i.e.,  how  the  various  elements  that 
are  perceived  at  the  earlier  level  are  integrated  and  comprehended).  At  Level  2  SA  the  various 
elements  perceived  in  the  situation  are  integrated  in  light  of  the  pilot’s  goal  and  the  requirements 
of  the  task.  At  this  level  the  pilot  forms  a  holistic  picture  of  the  environment  that  combines  the 
individual  elements  with  stored  knowledge  of  these  elements.  Level  3  SA  refers  to  the  pilot’s 
ability  to  predict  future  status  of  the  situational  elements  and  to  anticipate  the  requirements  of  the 
operating  system.  This  is  the  highest  level  of  SA  and  is  based  on  the  elements  of  both  Levels  1 
and  2. 
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Endsley  (1995)  proposed  a  theoretical  model  of  SA.  In  her  model  Endsley  describes  SA  as  a 
dynamic  mental  model  of  the  world  that  is  detennined  by  various  cognitive  factors  such  as 
attention,  working  memory,  and  prior  knowledge.  Unfortunately  Endsley’s  model  is  far  too 
complex  to  generate  testable  hypotheses  or  guide  experiments.  Also,  it  is  not  clear  within  the 
model  whether  measured  SA  is  the  behaviour  of  the  pilot  or  the  underlying  cognitive  state 
(Flach,  1995).  Furthermore,  Endsley  emphasizes  that  SA  as  a  cognitive  state  that  is  independent 
from  perfonnance.  Therefore,  performance  measures  often  play  a  small  role  in  cockpit 
evaluations  unless  specifically  defined  as  being  an  indirect  measure  of  SA.  In  summary, 
although  Endsley’s  model  has  had  considerable  impact  on  the  field  of  aviation  psychology,  it 
functions  more  as  a  way  of  describing  the  human-machine  interaction,  rather  than  as  a 
framework  for  developing  M&S  research. 


II.A.2  -  Measuring  Situation  Awareness 

In  combination,  Pew  (2000)  and  Vidilich  (2000)  proposed  five  broad  categories  of 
measurement  that  have  been  used  in  research  on  situation  awareness.  As  shown  in  Table  2.1, 
these  include  verbal  protocols,  awareness  queries,  subjective  assessments,  performance 
assessments,  and  situation  manipulations.  Pew  (2000)  provides  the  more  comprehensive 
taxonomy  of  SA  measures,  however,  Vidulich  (2000)  noted  that  researchers  have  often  inferred 
SA  from  performance,  rather  than  measuring  it  directly.  In  accord  with  the  proposed  CSE 
framework,  measuring  only  SA  or  only  workload  is  likely  to  be  restrictive.  The  overlap  between 
SA  and  performance  noted  by  Vidulich  supports  our  view  that  multiple  constructs  and  multiple 
measures  of  these  constructs  are  most  likely  to  provide  a  comprehensive  overview  of  the  human- 
machine  interaction. 
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Table  2.1 

Taxonomy  of  Measurement  for  Situation  Awareness  (after  Pew,  2000,  and  Vidulich,  2000) 


Our 

terminology 

Pew’s 

Terminology 

[Vidulich]3 

Description 

Verbal 

Protocols 

Verbal 

protocols 

Pilots  provide  online  or  immediately  retrospective 

think-aloud  descriptions  of  their  mental  processes 

Situation 

Manipulations 

Direct  system 

performance 

measures 

During  the  simulated  mission,  the  experimenter: 

(a)  introduces  either  an  anomaly,  or  some  kind  of 

subtle  scenario  manipulation  that  the  pilot  is  expected 

to  detect  if  they  have  good  SA 

(b)  introduces  disruptions  that  pilots  must  recover 

from  (e.g.,  freeze  the  simulation,  then  introduce  an 

offset  to  the  heading) 

(c)  introduces  anomalous  data  or  instrument  readings 

and  observes  pilots’  responses 

Awareness 

Queries* 

Direct 

experimental 

techniques 

[memory  probe 

measures] 

Pilots  respond  to  queries  or  probes  collected  when  a 

simulation  is  frozen  (e.g.,  best  known  method  is 

SAGAT;  Endsley  [2000]) 

Subjective 

Assessment* 

Subjective 

measures 

Self-assessments  (usually  after  the  mission  has 

finished),  expert  judgments  of  pilot  SA,  peer  ratings, 

supervisor  or  instructor  ratings 

Performance 

Assessment* 

Experimenter  makes  an  inference  about  effects  of  a 

technology  on  SA  by  evaluating  changes  in 

performance 

3  terminology  used  by  Vidulich  (2000)  if  it  is  different  than  that  used  by  Pew  (2000) 
*  categories  included  in  Vidulich  (2000)  meta-analysis. 
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II.A.3  -  Summary 

SA  represents  how  well  the  pilot  perceives  the  situational  elements  and  integrates  them  into 
a  coherent  understanding  of  the  situation.  Measurement  of  SA  can  show  how  a  well-designed 
system  supports  the  pilots’  cognitive  needs  whereas  a  poorly  designed  system  may  result  in 
pilots  lacking  a  coherent  understanding  of  their  surroundings  (Jones  &  Endsley,  1996).  However, 
although  SA  provides  important  insight  into  some  aspects  of  the  pilot-system  interaction,  other 
aspects  may  be  neglected.  For  example,  by  focusing  on  measuring  a  pilot's  understanding  of  the 
situation,  some  critical  aspects  of  behaviour  may  be  overlooked.  For  example,  perceptual-motor 
coordination  that  is  reflected  in  how  well  the  pilot  maintains  a  route  or  reacts  to  stimuli  in  the 
environment  is  an  important  part  of  the  overall  adaptation  to  the  requirements  of  the  cockpit 
system.  Thus,  measurement  of  SA  needs  to  be  supplemented  by  other  sources  of  information. 

The  proposed  CSE  framework  would  help  researchres  to  capture  more  of  the  variance  involved 
in  pilot-system  interaction  as  compared  to  measuring  a  single  construct.  Thus,  the  goal  of  the 
CSE  framework  is  to  guide  future  research  in  this  domain  based  on  the  assumption  that  multiple 
measures  of  three  central  constructs  (i.e.,  SA,  workload,  and  task-relevant  performance),  will 
give  a  reasonably  complete  perspective  on  how  any  given  technology  influences  human 
responses  in  the  cockpit. 

II.B.  -  Workload 

II.B.l  -  Defining  Workload 

The  concept  of  mental  workload  has  been  used  in  the  aviation  community  to  capture  the 
relations  among  task  difficulty,  limitations  in  pilots’  cognitive  abilities,  and  performance  (Flach 
&  Kuperman,  2001;  Wickens,  2001).  Workload  is  an  index  of  how  much  cognitive  effort  is 
required  by  pilots  as  they  interact  with  a  given  system.  Measuring  workload  is  particularly 
relevant  to  cockpit  research  where  the  challenge  is  to  reach  a  balance  between  the  quantity  of 
information  provided  and  the  pilots’  capacity  to  process  that  information.  In  general,  as  more 
effort  is  required  from  the  pilot  for  a  given  task,  fewer  cognitive  resources  are  available  for 
accomplishing  other  tasks.  High  workload  is  associated  with  emotional  stress,  fatigue,  and 
decrements  in  performance  (Hart  &  Wickens,  1990;  Tsang  &  Wilson,  1997). 
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Workload  has  been  defined  as:  “a  hypothetical  construct  that  represents  the  cost  incurred  by 
a  human  operator  to  achieve  a  particular  level  of  performance”  (Hart  &  Staveland,  1988,  p.  140). 
This  definition  of  workload  is  framed  within  the  resource  model  of  information  processing  such 
that  workload  represents  the  relation  between  the  availability  of  cognitive  resources  and  the 
demands  of  the  task  (Wickens,  1991).  Within  this  model,  performance  decrements  occur  when 
tasks  that  are  being  performed  simultaneously  require  the  same  cognitive  resources.  This 
resource-based  definition  of  workload  is  consistent  with  a  large  literature  on  the  relations 
between  mental  effort  and  performance  in  a  wide  range  of  cognitive  tasks.  In  aviation  research, 
however,  a  disadvantage  of  this  definition  of  workload  is  that  it  does  not  distinguish  between 
how  the  pilot  experiences  the  level  of  task  difficulty  and  the  actual  impact  it  has  on  perfonnance. 

In  the  domain  of  aviation,  researchers  have  placed  more  emphasis  on  the  pilot's  experience 
of  workload  (i.e.,  subjective  workload)  than  on  objective  workload  (i.e.,  performance  trade-offs) 
by  measuring  physiological  responses  and  asking  pilots  to  rate  their  perceived  workload  (Hart  & 
Wickens,  1990;  Wickens,  1999).  The  focus  on  how  pilots  experience  workload  reflects  the 
finding  that,  in  highly  trained  pilots,  high  workload  does  not  necessarily  reflect  poor 
performance  (Vidulich  &  Wickens,  1986;  Yeh  &  Wickens,  1988).  Nevertheless,  if  the  pilot 
experiences  a  task  as  demanding  (i.e.,  high  in  workload),  he  or  she  may  become  fatigued  and 
mission  effectiveness  may  be  compromised.  A  common  pattern  seen  in  workload  research  is  that 
performance  is  stable  for  a  long  time  and  then  suddenly  declines.  Initially,  researchers  assumed 
that  the  performance  decrement  reflected  a  sudden  increase  in  workload  that  exceeded  the  pilots’ 
cognitive  resources.  However,  subjective  ratings  and  physiological  measures  showed  that  pilots 
actually  experienced  a  steady  increase  in  workload  up  to  the  point  of  the  perfonnance  decrement 
(Andre,  2001). 

In  summary,  both  the  objective  workload  (i.e.,  trade-offs  between  performance  and  effort) 
and  subjective  workload  (i.e.,  the  pilots’  perception  of  workload)  are  important  aspects  of  the 
human-machine  interface  in  research  on  aviation.  However,  although  research  suggests  that 
workload  measures  are  a  valuable  tool  in  aviation  research,  they  have  not  been  used  consistently 
because  there  is  no  consensus  in  how  workload  should  be  measured  (Flach  &  Kuperman,  2001; 
Wickens,  2001). 
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II. B. 2  -  Measuring  workload 

Numerous  measures  have  been  proposed  to  evaluate  workload  in  the  cockpit.  The 
proposed  measures  of  workload  can  be  classified  into  four  broadly  defined  categories;  primary- 
task  measures,  secondary-task  measures,  physiological  measures,  and  subjective  ratings 
(Bortolussi,  Kantowitz  &  Hart,  1986;  Casali  &  Wierwille,  1984;  Hart  &  Wickens,  1990; 

Wickens  &  Hollands,  1999). 

Primary -task  measures.  Measuring  performance  on  the  primary  task  (i.e.,  the  task  required 
by  the  system  in  question)  allows  for  the  assessment  of  whether  the  task  causes  boredom  and 
hence  less  vigilance  over  time,  whether  perfonnance  is  stable  over  time,  and  at  what  point 
performance  breaks  down.  However,  performance  on  a  primary  task  is  rarely  used  to  evaluate 
workload  as  it  tends  not  to  co-vary  with  pilots’  experience  of  workload.  Therefore,  poor 
performance  may  or  may  not  reflect  demands  on  resources  (Yeh  &  Wickens,  1988).  Thus, 
primary  task  perfonnance  can  be  considered  a  baseline  measure,  but  it  is  only  directly  indicative 
of  workload  when  cognitive  resources  are  exceeded  and  perfonnance  begins  to  break  down. 

Secondary-  task  measures.  When  pilots  simultaneously  perform  two  tasks,  the  primary  task 
is  the  central  aviation  task  (e.g.,  hovering)  whereas  the  secondary  task  is  added  by  the  researcher 
to  reflect  the  availability  of  cognitive  resources.  For  example,  in  the  DVI  evaluation  reported  by 
Herdman  et  al.  (2001),  the  primary  task  was  to  complete  the  mission  whereas  the  secondary  task 
was  to  detect  the  auditory  and  visual  targets.  The  pilot  is  instructed  to  perfonn  as  well  as 
possible  on  the  primary  task  and  allocate  any  leftover  resources  to  the  secondary  task.  As  the 
primary  task  becomes  more  difficult,  fewer  resources  are  available  for  performing  the  secondary 
task  and  thus  the  focus  is  on  decrements  in  perfonnance  on  the  secondary  task.  Common 
secondary  tasks  used  to  evaluate  workload  are:  a)  a  rhythmic  tapping  task  where  the  pilot  must 
produce  a  finger  or  a  foot  tap  at  a  constant  rate,  b)  random  number  generation  where  the  pilot 
must  randomly  generate  numbers,  and  c)  reaction  time  to  probe  stimuli  (e.g.,  Herdman  et  al., 
2001). 

Secondary  task  measures  have  been  used  frequently  to  evaluate  workload.  However,  the 
method  has  some  limitations.  First,  when  the  primary  task  reaches  a  certain  level  of  difficulty  the 
pilots  may  simply  abandon  the  secondary  task.  Second,  research  on  workload  using  secondary 
measures  has  shown  that  different  types  of  secondary  measures  will  be  interfered  with 
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selectively  by  the  primary  measure.  For  example,  tapping  is  more  likely  to  interfere  with  a 
spatial  primary  task  than  with  a  verbal  primary  task  (Baddeley  &  Logie,  1999).  This  selective 
interference  means  that  the  workload  differences  caused  by  the  primary  task  may  be 
underestimated  if  the  primary  and  secondary  tasks  require  different  processing  resources  (see 
Hart  &  Wickens,  1990).  Third,  introducing  a  secondary  task  to  the  pilot  may  be  intrusive  for  the 
pilot.  Despite  these  limitations,  however,  secondary  task  measures  provide  a  very  useful 
objective  index  of  pilot  workload.  Furthennore,  the  use  of  the  dual-task  method  is  a  well- 
established  way  of  indexing  cognitive  demands  in  the  wider  literature  (Baddeley  &  Logie,  1999) 
and  thus  considerable  research  can  be  accessed  to  develop  and  interpret  the  results  of  workload 
research  in  the  aviation  field. 

Physiological  measures.  Physiological  measures  of  workload  include  heart  rate,  eye  blink 
rate,  pupil  diameter,  respiration  frequency,  blood  pressure,  and  electrical  activity  of  the  brain. 
Use  of  these  measures  is  based  on  the  assumption  that,  for  example,  an  increase  in  heart  rate  or 
respiration  reflects  a  concomitant  (but  not  necessarily  conscious)  increase  in  workload.  One 
advantage  of  physiological  measures  is  that  they  are  less  obtrusive  than  subjective  ratings  or 
secondary  task  measures.  Furthermore,  measurement  of  physiological  responses  provides 
information  about  the  pilots’  emotional  and  physical  activation  during  the  course  of  a  task  as 
well  as  their  processing  time  and  cognitive  load.  Researchers  have  shown  that  physiological 
measures  are  a  reliable  indication  of  workload  (Bortolussi,  Kantowitz  &  Hart,  1986;  Casali  & 
Wierwille,  1984).  However,  the  main  limitation  of  physiological  measures  is  that  they  are 
indirect  indices  of  how  the  pilot  actually  experiences  the  workload.  Furthennore,  physiological 
measures  may  not  relate  directly  to  performance. 1 

Subjective  evaluation.  Subjective  ratings  of  workload  have  been  used  most  frequently  in 
research  on  aviation.  They  have  the  advantage  of  directly  measuring  the  pilots’  experience  of 
workload  in  terms  of  cognitive  cost  and  attentional  resources  (Hart  &  Wickens,  1990).  Two 
questionnaire  measures  are  commonly  used  in  evaluating  the  pilots'  experience  of  workload,  the 
NASA  task  load  index  (NASA  TLX;  Hart  &  Staveland,  1988)  and  the  subjective  workload 
assessment  technique  (SWAT;  Reid  &  Nygren,  1988).  The  NASA  TLX  assesses  workload  on 


1  Although  visual  scanning  or  other  measures  of  behavior  are  sometimes  classified  as  “physiological  measures”,  in 
the  CSE  framework  these  would  be  considered  as  elements  of  “task-relevant  performance”.  Visual  scanning  is  a 
very  indirect  measure  of  physiological  processes. 
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five  7-point  scales;  mental  demand,  physical  demand,  temporal  demand,  perfonnance,  effort,  and 
frustration  level.  The  SWAT  assesses  workload  on  three  3-point  scales;  time  load,  mental  effort, 
and  stress. 

Subjective  ratings  of  workload  are  widely  used.  Subjective  ratings  are  relatively  easy  to 
administer  and  have  minimal  impact  on  pilots’  ongoing  behaviour  because  they  are  typically 
given  after  a  task  is  completed.  These  measures  are  limited,  however,  because  they  are  rarely 
used  to  index  online  workload  (i.e.,  workload  that  is  experienced  while  the  pilot  is  doing  the 
task)  and  because  they  are  subjective.  As  with  any  measure  that  requires  pilots  to  introspect  on 
their  behaviour,  there  are  always  questions  about  whether  such  measures  are  reliable  (i.e.,  are  the 
measures  consistent?  Can  pilots’  calibrate  their  responses  across  tasks?)  and  valid  (i.e.,  do  the 
responses  actually  reflect  workload?;  Wickens,  1999).  Furthermore,  rating  scales  are  limited  in 
that  they  will  only  tap  into  a  subset  of  factors  that  may  be  relevant  to  the  overall  experience  of 
workload  (Hart  &  Wickens,  1990).  More  research  in  which  subjective  measures  are  collected  in 
conjunction  with  objective  measures  is  needed  to  probe  the  validity  of  these  assessments. 

II. B. 3  -  Summary 

In  aviation  research,  assessment  of  workload  is  important  for  evaluating  the  adequacy  and 
feasibility  of  the  human-machine  interaction.  High  workload  may  reflect  a  poorly  designed 
system  that  puts  an  unnecessary  load  on  the  pilot.  For  example,  an  instrument  layout  in  the 
cockpit  that  requires  longer  gaze  time  by  pilots  than  an  alternative  layout  may  increase  pilot 
workload.  Similarly,  Herdman  et  al.  (2001)  found  that  the  addition  of  a  DVI  system  to  the 
CHI 46  cockpit  resulted  in  an  increase  in  the  workload  of  the  flying  pilot  (as  indexed  by  the 
secondary  task  performance).  Hence,  it  is  crucial  to  measure  pilot  workload  in  any  technology 
evaluation.  Although  four  categories  of  workload  measurement  have  been  described  (i.e., 
primary-task  measures,  secondary-task  measures,  physiological  measures  and  subjective  ratings), 
multiple  measures  have  rarely  been  used  in  the  same  studies.  Furthermore,  workload  measures 
have  not  been  used  consistently  in  cockpit  research  (Wickens,  2001).  As  with  performance  and 
situation  awareness,  converging  measures  of  workload  are  most  likely  to  provide  a 
comprehensive  understanding  of  the  impact  of  technology  on  the  cockpit  activity. 
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Executive  Summary 

Recent  advances  in  both  knowledge  of  human  psychology  and  the  capabilities  of 
simulation  environments  support  a  greater  role  for  modeling  and  simulation  (i.e.,  M&S)  in 
human  factors  research,  system/equipment  acquisition,  and  in  the  development  of  training 
programs.  As  part  of  the  Tactical  Aviation  Mission  System  Simulation  (TAMSS)  initiative,  the 
Centre  for  Applied  Cognitive  Research  (CACR)  at  Carleton  University  has  proposed  a  Cognitive 
Systems  Engineering  (CSE)  framework  to  be  used  as  a  guide  for  conducting  and  interpreting 
evaluations  in  M&S  programs.  The  present  document  includes  a  report  on  an  initial  experiment 
conducted  at  the  CACR  using  the  CSE  framework.  In  accord  with  the  CSE  framework,  the 
results  of  the  experiment  demonstrate  the  importance  of  collecting  converging  measures  of  pilot 
performance,  workload,  and  situation  awareness.  The  present  document  also  includes  an 
overview  of  the  development  and  current  capability  of  the  CACR  simulator  research  facility. 
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Situation  Awareness,  Performance,  and  Workload  in  Simulated  Flight: 

Head-Up  Display  (HUD)  versus  Head-Down  Display  (HDD) 

This  document  has  two  main  sections.  In  Section  I,  a  report  is  given  on  the  initial 
experiment  in  the  TAMSS  SA  program.  In  this  experiment,  the  Cognitive  Systems  Engineering 
(CSE)  framework  that  has  been  proposed  by  the  Carleton  University  Centre  for  Applied 
Cognitive  Research  (CACR)  was  used  to  evaluate  the  human-machine  interface  in  two 
conditions,  head-up  versus  head-down  displays.  In  accord  with  the  CSE  framework,  the 
experiment  demonstrates  the  utility  of  collecting  converging  measures  of  pilot  performance, 
workload,  and  situation  awareness  (SA)  in  modelling  and  simulation  (M&S)  efforts.  In  Section 
II,  an  overview  is  given  of  the  development  and  current  capability  of  the  CACR  simulator 
research  facility. 

Section  I:  Experiment 


1.1  Introduction 

1.1.1  Goals 

The  goals  of  this  experiment  were  to  (a)  provide  a  preliminary  evaluation  of  the 
Cognitive  System  Engineering  (CSE)  framework  (see  Annex  A)  proposed  by  the  CACR  for 
evaluating  the  impact  of  novel  technology  on  aircrew  in  the  CHI 46  Griffon  helicopter  and  (b) 
develop  and  test  the  experimental  capabilities  of  the  Carleton  University  simulator  facility. 

In  this  experiment,  pilots  flew  a  series  of  simplified  recce-type  missions  while  wearing  a 
HMD.  Two  primary  conditions  were  compared:  HUD  versus  HDD.  In  the  HUD  condition,  the 
HMD  was  equipped  with  HUD  symbology  showing  primary  flight,  power,  and  navigation 
information.  The  HUD  symbology  was  derived  from  the  CHI 46  Griffon  LATEF  II  HUD.  In  the 
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HDD  condition,  the  HMD  was  not  equipped  with  HUD  symbology.  Instead,  pilots  were 
required  to  look  under  the  HMD  to  acquire  the  requisite  information  from  the  head-down 
instrument  panel.  The  HDD  condition  is  similar  to  that  faced  by  CHI 46  Griffon  pilots  using 
Night  Vision  Goggles  (NVGs):  when  the  NVGs  are  not  equipped  with  a  HUD,  the  pilots  must 
look  under  the  goggles  to  read  information  from  the  instrument  panel. 

The  experiment  was  designed  to  sample  all  three  dimensions  of  behaviour  outlined  in  the 
CSE  framework:  task-relevant  performance,  workload,  and  situation  awareness.  Furthermore, 
the  main  thrust  of  the  CSE  framework  is  to  provide  converging  measures,  allowing  researchers 
to  give  an  overall  perspective  of  performance  that  does  not  rely  solely  on  a  single  construct  or 
single  method  of  measurement.  Both  subjective  and  objective  measures  of  each  dimension  were 
developed.  Although  the  TAMSS  SA  project  is  focused  on  situation  awareness,  we  contend  that 
assessing  SA  in  isolation  will  not  provide  sufficient  evidence  to  allow  for  good  decisions  in  the 
modelling  and  simulation  process. 

1.1.2  Overview  of  measures 

Task-relevant  Performance  -  In  this  experiment,  pilots  flew  simplified  recce-type 
missions.  On  each  mission,  they  initially  took  off  from  a  base,  located  centrally  in  the  terrain. 
They  were  then  directed  to  find  a  series  of  waypoints  (towers  placed  in  the  terrain)  by  an 
experimenter  who  gave  them  heading  values.  During  the  course  of  the  recce,  pilots  were 
instructed  to  provide  reports  of  (sitreps)  any  and  all  activity  (in  the  air  or  on  the  ground).  Each 
scenario  was  populated  with  a  variety  of  objects  such  as  fighter  jets,  rotary- wing  aircraft,  and 
land  vehicles.  These  objects  varied  in  visibility,  but  all  were  visible  for  a  minimum  of  2  to  3 
seconds  in  daylight  conditions.  Although  performance  was  measured  exhaustively,  our  focus  in 
this  report  is  on  deviations  from  assigned  speed  and  altitude  values.  Subjective  ratings  of 
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performance  and  of  task  difficulty  were  collected  after  each  mission. 

Workload  -  Workload  was  assessed  through  the  presentation  of  auditory  and  visual 
probes.  Both  the  latency  to  the  probes  and  the  percentage  of  probes  missed  was  measured. 
Subjective  ratings  of  workload,  globally  and  during  specific  legs  of  the  recce,  were  also 
collected. 

Situation  Awareness  -  The  objective  index  of  situation  awareness  was  the  percentage  of 
objects  that  pilots  missed  during  each  mission.  After  each  mission,  pilots  also  rated  their 
perceived  awareness  overall  and  for  specific  flight  parameters  (speed,  altitude,  and  heading),  and 
for  activity  in  the  environment. 

1.1.3  Predictions 

The  pilots  who  volunteered  for  this  study  were  all  highly  experienced  with  rotary-wing 
aircraft.  However,  none  of  the  pilots  had  experience  with  the  HUD  symbology.  Thus, 
differences  due  to  familiarity  of  the  standard  instruments  versus  the  HUD  symbology  would  be 
expected,  especially  where  the  infonnation  conveyed  by  the  symbology  was  in  a  very  different 
form  from  that  of  the  instruments  (e.g.,  speed  on  a  dial  versus  speed  in  numbers).  Furthermore, 
although  HUDs  have  been  shown  to  be  effective  in  specific  tasks  such  as  controlling  flight  path 
and  altitude  (e.g.,  see  Fadden,  Ververs,  &  Wickens,  2001;  Martin-Emerson  &  Wickens,  1997; 
Wickens  &  Long,  1995),  there  are  a  number  of  simulator-based  studies  suggesting  that  pilots 
may  focus  or  ‘cognitively  tunnel’  their  attention  on  HUD  symbology  (Brickner,  1989;  Fischer, 
Haines,  &  Price,  1980;  Foyle,  Stanford,  &  McCann,  1991,  Wickens  &  Long,  1995).  Cognitive 
tunneling  may  cause  pilots  to  miss  potentially  critical  events  in  the  external  scene.  For  example, 
Fisher  et  al.  found  that  in  a  simulated  landing  task,  pilots  were  less  likely  to  detect  a  runway 
incursion  (e.g.,  a  vehicle  driven  onto  a  runway)  when  they  were  using  a  HUD  than  when  they 
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were  using  a  conventional  head-down  display.  In  the  present  research  the  object  detection  task 
was  more  naturalistic  than  those  used  in  previous  simulator-based  studies  of  cognitive  tunneling. 
Because  attentional  capture  is  a  potential  side  effect  of  HUDs,  it  was  predicted  that  pilots  who 
are  not  familiar  with  HUDs  would  show  evidence  for  cognitive  tunnelling. 

1.2  Method 


1.2.1  Participants 

Four  male  pilots  participated  in  this  experiment.  They  ranged  in  age  from  37  to  50  years. 
The  pilots  had  between  10  and  29  years  experience,  with  1800  -  4800  hours  total  flight  time  and 
780  -  1200  total  hours  in  the  Griffon.  None  had  any  experience  using  either  fixed  panel  or 
HMD  HUDs.  Thus,  all  were  seasoned  pilots  but  novice  HUD  users. 


1.2.2  Design 

The  central  independent  variable  in  this  experiment  was  a  comparison  of  the  HUD  versus 
the  HDD.  The  objective  and  subjective  measures  of  the  three  core  constructs,  situation 
awareness,  workload,  and  task-relevant  perfonnance  are  shown  in  Table  1.1. 


Table  1.1:  Description  of  Dependent  Measures 

Type  of  Measure 

CSE  Domain 

Objective 

Subjective 

Task-Relevant 

Performance 

Deviation  from  assigned  speed, 
altitude,  and  heading 

Ratings  of  performance  and 
task  difficulty 

Situation 

Awareness 

Percentage  of  objects  detected  in 
the  external  scene 

Ratings  of  SA  (specific  and 
global) 

Workload 

Auditory  and  Visual  probe 
detection  (percentage  detected  and 
mean  detection  latencies) 

Ratings  of  workload 
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1.2.3  Materials 

Questionnaires  -  The  subjective  measurements  of  situation  awareness,  workload,  and 
performance  were  conducted  at  the  end  of  each  mission  and  at  the  end  of  the  experiment.  Pilots 
rated  these  variables  on  a  number  of  scales  (refer  to  Annex  B).  In  addition,  before  starting  the 
experiment,  pilots  completed  a  background  questionnaire  (see  Annex  B),  which  included 
questions  about  the  number  of  tactical,  Griffon,  and  HUD  (heads-up  display)  flying  hours  they 
had  logged. 

Development  of  Mission  Scenarios  -  The  scenarios  for  the  experiment  were  developed 
using  the  input  from  a  subject-matter  expert  (SME)  provided  by  DND  to  set  up  realistic  missions 
for  the  pilots  to  complete.  Numerous  entities  were  added  to  the  terrain  database,  creating 
scenarios  that  would  allow  the  experimenters  to  take  measures  of  Situation  Awareness, 
Workload,  and  Task-relevant  performance.  The  following  is  a  description  of  the  database  used 
and  the  additions  made  to  the  external  scene  for  the  purposes  of  the  experiment. 

Terrain  database  -  The  landscape  database  was  a  Virtual  Reality  model  of  a  10  km  by  10 
km  section  of  CFB  Gagetown,  NB.  The  database  contained  a  number  of  fixed,  pre-determined 
geographical  features  (river,  hills,  forest)  and  man-made  elements  (barracks,  various  military 
installations,  roads,  and  the  flight  base).  Various  entities,  both  moving  and  stationary,  were 
added  to  the  terrain  database  to  create  a  number  of  mission  scenarios  (see  below).  Some  of  the 
entities  were  fixed  navigation  landmarks,  which  allowed  pilots  to  follow  pre-detennined  flight 
paths  as  instructed  by  the  experimenters.  Other  entities  added  to  the  terrain,  such  as  military 
vehicles  and  armaments,  were  used  to  assess  pilots’  situation  awareness  during  missions. 

Navigation  landmarks  -  Eight  fixed  objects  were  used  as  markers  to  indicate  the 
waypoints  that  made  up  specific  flight  routes.  They  were  placed  at  the  four  corners  of  the 
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database  terrain  and  midway  between  the  corners  on  each  edge,  roughly  0.5  km  in  from  the 
edges.  The  markers  themselves  were  10  m  tall  white  rectangular  parallelepipeds  with  a  square 
base  (i.e.,  tall  white  narrow  boxes)  that  were  visible  up  to  5  km  away.  These  markers  were 
chosen  because  of  their  high  visibility  in  the  HMD  goggles.  They  were  inserted  into  the  terrain 
database  using  the  STAGE  program. 

Objects  used  for  assessing  pilot  SA  -  A  number  of  objects  were  included  in  the  simulation 
in  order  to  provide  pilots  with  entities  to  report  during  their  missions.  Objects  were  inserted  and 
controlled  using  the  STAGE  software,  with  the  exception  of  one  wrecked  helicopter,  which  was 
inserted  using  the  VEGA  environment.  The  objects  included  (a)  two  moving  formations  of  three 
armoured  ground  cars,  (b)  three  stationary  pieces  of  artillery  (Howitzer  guns),  (c)  four  grounded 
CH-149  Cormorant  helicopters,  (d)  one  wrecked  CH-149,  (e)  two  CH-149s  flying  in  small  loop 
formations,  (f)  two  hovering  CH-146  Griffon  helicopters,  (g)  one  formation  of  four  CF-18s 
flying  in  a  wide  formation  across  a  large  portion  of  the  terrain,  and  (h)  one  C-130  Hercules  fixed- 
wing  transport  aircraft  flying  a  slow,  elongated  loop  pattern  that  cut  across  the  whole  width  of 
the  database  terrain,  roughly  five  kilometres  from  the  southern  edge  of  the  terrain.  All  vehicles 
were  placed  so  that  they  were  on,  or  intersected,  the  paths  that  pilots  flew  in  their  missions. 
Hence,  most  vehicles  were  close  to  the  edges  or  on  the  diagonals  of  the  square  formed  by  the 
database,  and  were  either  on  the  ground  or  at  a  fairly  low  altitude  (below  300  feet).  The  CF-18s 
and  the  C-130  flew  relatively  slow  and  wide  trajectories  that  intersected  the  pre-planned  mission 
routes  at  fairly  regular  intervals.  All  entities  were  scaled  to  their  normal  size  relative  to  the 
database. 

Mission  Scenarios  -  Two  separate  terrain  databases  were  used  in  the  experiments.  Each 
database  contained  the  same  geographical  features,  buildings,  waypoint  markers,  and  entities, 
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and  differed  only  in  that  the  entities  had  different  locations  and  trajectories  in  each  database. 

Each  pilot  flew  four  missions  in  each  terrain  (one  terrain  on  the  first  day  of  their  participation, 
the  other  on  the  second  day),  for  a  total  of  eight  missions  per  pilot.  Missions  were  limited  to  four 
per  terrain  to  minimize  the  likelihood  that  pilots  would  rely  on  memory  to  report  visual  contacts 
with  SA  assessment  objects. 

Each  mission  consisted  of  flight  legs  (defined  as  a  trajectory  between  two  successive 
waypoint  markers)  arranged  in  a  different  order.  The  flight  legs  were  sequenced  such  that  (1) 
each  waypoint  was  reached  once  per  mission,  (2)  all  the  SA  assessment  objects  were  included  on 
the  path  and  distributed  approximately  equally  between  the  legs  of  the  mission,  and  (3)  the  legs 
constituted  a  continuous  path  starting  and  ending  at  the  base.  Consequently,  two  successive  legs 
could  either  be  collinear  or  at  an  angle  to  each  other  (either  45°  or  90°  depending  on  whether  both 
legs  were  on  the  edges  of  the  database,  or  one  was  on  a  diagonal  between  a  comer  waypoint 
marker  and  the  base).  Thus  each  mission  was  defined  as  a  specific  path  visiting  all  eight 
waypoints,  and  was  detennined  prior  to  starting  the  experiments.  Examples  of  the  mission  routes 
and  positions  of  the  vehicles  are  provided  in  Annex  C. 

1.2.4  Procedure 

Upon  arrival,  the  pilots  were  provided  with  some  information  about  the  experiment  and 
were  given  an  overview  of  the  two-day  schedule.  Pilots  then  completed  an  informed  consent  and 
the  background  questionnaire.  Following  the  information  session,  the  pilots  flew  three  practice 
sessions  before  beginning  the  first  of  eight  experimental  sessions.  The  practice  sessions  consisted 
of  a  simulated  flight  using  the  full  OTW  scene,  a  second  flight  using  the  HMD  without  the  HUD, 
and  a  final  practice  mission  using  the  HMD  with  the  HUD.  Prior  to  beginning  the  practice 
flights,  the  pilots  were  briefed  on  the  functionalities  of  the  HUD  symbology.  During  all  three 
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practice  sessions  the  pilots  were  required  to  report  on  any  activity  (e.g.,  aircraft,  ground  vehicles) 
seen  during  the  mission.  After  a  brief  break,  the  pilots  flew  the  first  experimental  mission.  Their 
task  during  each  experimental  mission  was  to  follow  the  designated  flight  path  while  maintaining 
the  assigned  speed  and  altitude.  There  were  also  to  report  any  activity  occurring  on  the  ground 
or  in  the  air  throughout  the  mission. 

During  Mission  1,  Pilots  1  and  2  flew  using  the  HMD  without  the  HUD.  For  Mission  2, 
they  flew  using  the  HMD  with  the  HUD.  For  the  remaining  six  missions,  Pilots  1  and  2 
alternated  between  flying  with  the  HUD  and  flying  without  the  HUD.  In  contrast,  Pilots  3  and  4 
flew  their  first  mission  using  the  HMD  with  the  HUD  and  flew  the  second  mission  using  the 
HMD  without  HUD.  Pilots  3  and  4  also  alternated  between  the  HUD  and  no-HUD  conditions  for 
the  remaining  missions. 

During  the  final  HUD  mission,  the  experimenters  froze  the  heading  tape  for 
approximately  15  -30  seconds  to  test  “freezing  the  instruments”  as  a  possible  measure  of  SA  in 
future  experiments. 

Workload.  In  half  the  missions  (two  with  the  HUD  and  two  without  the  HUD)  the  pilots 
were  instructed  to  respond  as  quickly  as  possible  to  auditory  and  visual  probes.  The  visual  probe 
consisted  of  a  light  that  appeared  for  500  ms  at  the  center  of  the  pilot’s  field-of-vision  (FOV)  at 
random  time  intervals  (i.e.,  every  15  seconds  plus  or  minus  20%  during  the  mission).1  The 
auditory  probe  was  a  randomly  presented  tone,  also  presented  briefly  (500  ms)  every  15  seconds 
(i.e.,  plus  or  minus  20%).  Upon  detecting  an  auditory  or  visual  probe,  pilots  were  to  respond  by 
pushing  a  button  on  the  cyclic  as  quickly  as  possible.  Only  one  type  of  probe  (visual  or  auditory) 
was  presented  during  a  particular  mission  and  the  pilot  was  informed  about  the  type  probe  before 
each  mission. 
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In  total,  each  pilot  completed  eight  missions,  four  with  the  HUD  and  four  with  the  HDD. 
Four  experimental  missions  were  flown  on  day  one  with  the  other  four  completed  on  day  two. 
Each  scenario  began  at  the  base  (in  the  center  of  the  terrain).  Pilots  were  instructed  to  take  off 
from  the  base,  and  then  head  in  a  direction  indicated  by  the  experimenter.  The  pilot  was  directed 
to  a  waypoint  marker  with  compass  directions.  Once  the  pilot  had  visually  identified  the 
waypoint,  they  were  given  a  new  heading  and  were  instructed  to  take  an  inside  turn  (if  possible) 
around  the  tower  and  go  to  the  new  heading.  A  sample  flight  scenario  through  the  terrain  is 
shown  in  Annex  C. 

Pilots  were  instructed  to  climb  to  a  height  of  200  m  and  maintain  airspeed  of  80  knots 
over  the  course  of  the  mission.  The  main  goal  of  the  mission,  however,  was  for  the  pilots  to 
verbally  report  the  presence  of  objects  in  the  environment.  In  each  experimental  condition,  the 
defined  heading  directions  were  communicated  to  the  pilots  by  one  of  the  experimenter  using  the 
simulator  intercom  system.  A  second  experimenter  recorded  the  appearance  of  the  objects, 
whether  or  not  the  pilot  reported  seeing  the  objects,  and  the  approximate  distance  between  the 
helicopter  and  the  object  when  the  pilot  reported  it.  Whenever  the  pilot  spotted  an  object,  they 
were  to  report  it  immediately  and  to  provide  detail  that  they  determined  was  relevant  (e.g.,  type 
of  object,  direction  object  was  heading  etc.).  The  number  of  missed  objects  was  also  recorded. 
The  two  experimenters  viewed  two  computer  screens  located  behind  the  pilot.  One  showed  the 
actual  scene  the  pilots  saw  at  any  given  point  in  time,  and  the  other  showed  the  location  of  the 
ownship  on  a  map  of  the  database.  Thus,  both  experimenters  had  full  access  to  the  scene  being 
viewed  by  the  pilot  and  could  monitor  pilot  activities  and  the  movement  of  the  aircraft.  The 
experimenters  reminded  the  pilots  of  these  headings  when  they  deviated  from  them.  Upon 
reaching  each  waypoint  a  new  heading  (north,  south,  west,  east)  was  given  for  the  next  waypoint. 

1  Due  to  a  failure  to  log  experimental  data,  complete  visual  probe  data  is  not  available  for  two  pilots. 
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When  the  pilots  had  successfully  completed  the  mission,  they  were  instructed  to  return  to  the 
base.  Upon  reaching  the  base,  the  simulation  was  frozen  and  the  experimental  session  ended. 

Each  mission  took  approximately  20  minutes  to  complete.  After  each  mission,  the  pilots 
were  asked  to  fill  out  a  questionnaire  that  concerned  their  awareness  of  objects,  events,  and 
instrument  readings  during  the  flight  (see  Annex  B).  While  one  pilot  was  filling  out  his 
questionnaire,  the  second  pilot  flew  his  next  mission.  At  the  end  of  the  second  day  the  pilots 
were  asked  to  fill  out  a  final  questionnaire  asking  about  their  experiences  with  the  HUD  versus 
the  HDD  condition. 
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1.3  Results 

1.3.1.  Missions 

The  main  focus  of  data  analyses  was  a  comparison  of  the  HUD  vs.  HDD  conditions. 
Objective  measures  of  task-relevant  performance  including  average  deviations  for  speed  and 
altitude  were  taken  during  each  mission.  The  objective  measure  of  SA  was  the  percentage  of 
objects  missed  during  each  mission.  The  objective  measures  of  workload  were  the  percentage  of 
auditory  and  visual  probes  detected,  and  the  average  speed  with  which  pilots  responded  to  these 
probes.  The  average  levels  of  performance  for  each  of  the  objective  measures  are  shown  in 
Table  1.2.  Differences  between  the  HUD  and  HDD  conditions  were  tested  with  directional  t- 
tests  (i.e.,  based  on  previous  research,  the  HUD  condition  was  predicted  to  result  in  worse 
performance  than  the  HDD  condition). 


Table  1.2:  Objective  Measures  of  Performance,  Workload,  and  Situation  Awareness 


Flight  Condition 

HUD 

HDD 

M 

SD 

M 

SD 

t(3) 

Situation  Awareness 

%  Objects  Missed  (all) 

48.5 

14.0 

37.5 

11.9 

2.82* 

•  In  air 

46.9 

15.6 

31.5 

14.3 

14.38** 

•  On  ground 

51.4 

16.2 

45.9 

9.9 

0.63 

Workload 

Auditory  %  Errors 

17.6 

16.0 

9.9 

8.4 

1.85f 

Visual  %  Errors3 

37.2 

41.7 

27.3 

17.9 

- 

Visual  RT  (ms)a 

980 

313 

921 

183 

- 

Auditory  RT  (ms) 

647 

147 

585 

54 

0.95 

Performance  (RMSD) 

Speed 

6.7 

1.9 

6.6 

0.3 

0.18 

Altitude 

23.4 

6.9 

24.9 

4.9 

-0.61 

“Based  on  two  pilots  only.  f  p  <  .10,  *  p  <  .05,  **  p  <  .01 

The  most  dramatic  result  in  Table  1.2  was  that  the  pilots’  objective  situation 

awareness 
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was  worse  in  the  HUD  than  in  the  HDD  condition.  Pilots  missed  more  objects  in  the  HUD 
condition  than  in  the  HDD  condition,  with  the  airborne  objects  (i.e.,  FI 8s,  Hercules,  and  other 
helicopters)  showing  the  greater  effect.  These  SA  differences  between  the  HUD  vs.  HDD 
conditions  do  not  appear  to  be  due  to  relative  duration  or  relative  visibility  of  the  various  objects, 
as  there  was  a  considerable  range  for  both  airborne  and  ground-based  objects. 

Workload  also  was  higher  in  the  HUD  than  in  the  HDD  condition,  such  that  pilots  missed 
more  of  the  auditory  and  visual  probes  (i.e.,  tones  and  light  flashes)  when  they  were  using  HUD 
symbology  than  when  they  were  using  the  instruments.  Similarly,  their  response  times  to  the 
probes  were  also  slower  in  the  HUD  than  in  the  HDD  condition.  Although  none  of  the  workload 
differences  were  statistically  significant,  the  consistent  pattern  of  differences  and  the 
substantially  larger  amount  of  variability  shown  in  the  HUD  condition  suggest  that  the  measures 
were  sensitive  to  workload  differences  but  that  there  was  considerable  cross-pilot  variability. 

In  contrast  to  the  workload  and  SA  measures,  task-relevant  performance  (i.e.,  average 
RMSD  deviations  from  speed  and  altitude)  did  not  vary  across  conditions  (see  Table  1.2).  Pilots 
were  able  to  maintain  flight  parameters  within  reasonable  boundaries  in  both  HUD  and  HDD 
conditions.  This  pattern  (differences  for  SA  and  workload  but  not  for  performance)  contrasts 
with  that  found  for  the  subjective  measures,  as  described  below.  It  appears  that  difficulties  with 
the  HUD  symbology  resulted  in  different  subjective  and  objective  assessments. 

After  each  mission,  pilots  completed  a  subjective  questionnaire  in  which  they  rated  their 
performance,  workload,  and  situation  awareness.  The  mean  ratings  on  each  question  are  shown 
in  Tables  1.3,  1.4,  and  1.5.  As  shown  in  Table  1.3,  objective  and  subjective  measures  showed 
moderate  convergence  in  this  experiment.  Pilots  rated  their  overall  SA  as  somewhat  worse  in  the 
HUD  than  in  the  HDD  condition,  in  accord  with  their  actual  perfonnance  on  the  object  detection 
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task.  They  also  rated  their  awareness  of  airspeed  and  of  activity  on  the  ground  as  worse  with  the 
HUD.  However,  they  did  not  perceive  their  performance  on  detecting  activity  in  the  air  as  worse 
with  the  HUD,  suggesting  that  their  subjective  access  to  specific  aspects  of  their  SA  was  low. 
These  results  do  suggest  that  the  questionnaire  is  sensitive  to  variations  in  perceived  SA. 


Table  1.3: 

Subjective  Measures  of  Situation  Awareness 

Flight  Condition 

HUD 

HDD 

M 

SD  M 

SD 

t(3) 

Overall  Situation  Awareness 

5.0 

0.4 

5.4 

0.5 

-2.05f 

Aviation: 

Heading 

4.8 

0.3 

4.9 

0.1 

-0.25 

Airspeed 

3.4 

0.4 

4.0 

0.3 

-3.06* 

Altitude 

4.3 

0.8 

4.6 

0.4 

-0.85 

Attitude 

4.8 

0.6 

4.8 

0.5 

0.40 

Aircraft  Systems 

4.2 

1.0 

4.3 

0.6 

-0.44 

OTW  Events: 

Activity  on  the  ground 

4.4 

1.0 

4.7 

0.8 

-2.43* 

Activity  in  the  air 

4.9 

0.6 

4.9 

0.6 

-0.20 

Environmental  Events 

5.2 

0.5 

5.0 

0.8 

0.70 

Spatial  Orientation 

5.5 

0.3 

5.4 

0.5 

0.13 

Note:  Situation  Awareness; 

1  =  low;  4 

=  moderate;  7  =  high 

f  p  <  .10,  *  p  <  .05,  **  p<  .  01 


Ratings  of  workload  are  shown  in  Table  1.4.  Subjectively,  pilots  did  not  rate  their 
workload  as  different  in  the  HUD  versus  the  HDD  condition.  This  contrasts  to  the  trend  for  the 
objective  measures  of  workload,  which  suggest  that  workload  was  greater  with  the  HUD. 
Instead,  pilots  rated  their  performance  as  worse  with  the  HUD  than  with  the  instruments,  as 
shown  in  Table  1.5.  They  indicated  that  their  performance  on  maintaining  speed,  altitude,  cross¬ 
checking  their  instruments,  and  using  information  from  the  external  scene  was  worse  with  the 
HUD  than  with  the  HDD.  This  latter  rating,  in  particular,  suggests  that  pilots  were  aware  that 
they  were  less  able  to  fulfill  all  the  demands  of  the  missions  with  the  HUD,  but  that  a  direct 
question  about  perfonnance  was  more  sensitive  to  these  differences  than  questions  about  SA  or 
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workload.  One  possible  explanation  of  these  results  is  that  pilots  are  more  able  to  evaluate  their 
performance  (because  they  have  direct  experience  of  it)  than  to  evaluate  workload  or  SA,  which 
are  hypothetical  constructs  that  may  have  different  subjective  meanings  to  different  individuals. 


Table  1.4: 

Subjective  Measures  of  Workload 

Flight  Condition 

t(3) 

HUD 

HDD 

M 

SD 

M 

SD 

At  waypoints 

3.2 

0.4 

3.2 

0.7 

0.12 

Between  Waypoints 

3.6 

0.6 

3.9 

0.8 

-1.31 

When  Reporting  Targets 

4.0 

0.8 

3.9 

0.8 

0.63 

Overall 

3.5 

0.6 

3.6 

0.8 

-0.73 

Note:  Workload;  1  =  low;  4  =  moderate;  7  =  high 
f  p  <  .10,  *  p  <  .05,  **  p<  .  01 


Table  1.5:  Subjective  Measures  of  Performance 

_ Flight  Condition _ 


HUD  HDD 


M 

SD 

M 

SD 

t(3) 

Finding  Waypoints 

5.4 

0.8 

5.7 

0.5 

-0.91 

Maintaining  Heading 

5.1 

0.6 

5.1 

0.5 

.00 

Maintaining  Speed 

3.4 

0.6 

4.2 

0.4 

-5.38** 

Maintaining  Altitude 

3.2 

0.5 

4.7 

0.2 

-4.61** 

Cross  Checking  Instruments 

4.2 

0.6 

5.2 

0.1 

-3.93** 

Using  Information  from  the 
External  Scene 

4.7 

0.6 

5.2 

0.4 

-2.47* 

Note:  Performance  Ratings;  1  =  very  poor;  4  =  adequate;  7  =  very  good 
f  p  <  .10,  *  p  <  .05,  **  p  <  .01 


This  interpretation  of  the  differential  sensitivity  of  the  less  direct  subjective  assessments 
(i.e.,  SA  and  workload)  versus  more  direct  subjective  assessments  (i.e.,  performance)  is 
supported  by  the  results  of  the  subjective  ratings  of  task  difficulty  and  difficulty  of  using  the 
available  information,  as  shown  in  Table  1.6.  Pilots  indicated  that  they  found  it  more  difficult  to 
maintain  speed  and  altitude  with  the  HUD  than  with  the  HDD.  They  also  found  it  more  difficult 
to  use  infonnation  about  speed  from  the  HUD  than  from  the  HDD.  These  differences  in  task 
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difficulty  and  useability  may  be  due  to  the  substantial  changes  in  the  representation  of  speed  and 
altitude  between  the  two  presentation  inodes.  On  the  familiar  instrument  panel,  speed  is 
represented  as  a  dial  (analogue)  whereas  on  the  HUD,  speed  is  represented  digitally  (numerical 
display).  Similarly,  altitude  is  represented  as  a  dial  on  the  HDD  panel,  but  as  a  combined  bar 
and  numerical  index  on  the  HUD.  Seemingly  trivial  differences  in  how  stimuli  are  represented 
can  have  a  significant  influence  on  how  those  stimuli  are  processed,  particularly  when  one 
representation  is  very  familiar  and  the  other  is  unfamiliar  or  unusual  because  different  mental 
codes  can  be  activated  by  different  inputs.  Consistently,  the  pilots  rated  their  perfonnance,  SA, 
and  the  perceived  difficulty  of  processing  speed  and  altitude  as  greater  in  the  HUD  than  in  the 
HDD  condition.  These  findings  indicate  that  the  questionnaire  was  sensitive  to  these  variations 
in  the  technology. 


Table  1.6:  Subjective  Ratings  of  Task  Difficulty  and  Difficulty  of  Using  Available 
_ Information _ 

_ Flight  Condition _ 


HUD  HDD 


M 

SD 

M 

SD 

t(3) 

Task  Difficulty  Ratings 

Finding  Waypoints 

2.2 

0.6 

2.4 

0.8 

-1.04 

Maintaining  Heading 

2.9 

0.5 

3.2 

0.5 

-1.84 

Maintaining  Altitude 

4.8 

0.5 

3.9 

0.4 

8.23** 

Maintaining  Speed 

4.6 

0.7 

3.7 

0.9 

5.02** 

Cross  Checking  Ins 

3.7 

0.7 

3.3 

0.9 

0.71 

Use  External  Scene 

2.8 

0.7 

2.9 

1.0 

-0.04 

Useability  Ratings 

Heading 

2.7 

0.5 

3.2 

0.9 

-1.33 

Altitude 

3.6 

1.4 

3.0 

0.7 

1.31 

Speed 

4.9 

0.4 

3.4 

0.5 

3.51** 

Attitude 

3.7 

1.3 

3.2 

0.7 

1.65f 

External  Scene 

2.9 

1.0 

2.8 

0.7 

0.37 

Note:  Difficulty  ratings:  l=very  easy,  4  =  moderate,  7  =  very  hard 


f  p  <  .10,  *  p  <  .05,  **p  <  .01 

In  contrast  to  the  difficulties  that  pilots  reported  with  speed  and  altitude  information,  they 
tended  to  rate  maintaining  heading  as  somewhat  easier  with  the  HUD  than  with  the  HDD,  despite 
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the  unfamiliar  representation  of  heading  (i.e.,  heading  tape)  in  the  HUD  versus  the  familiar 
format  (i.e.,  compass)  in  the  HDD.  The  trend  suggests  that  the  questionnaire  is  sensitive  to 
positive  as  well  as  negative  differences  in  representations  of  instruments. 

Finally,  as  noted  in  the  Procedure  section,  the  heading  infonnation  was  frozen  for 
approximately  15-30  seconds  for  pilots  sometime  during  their  last  HUD  mission.  None  of  the 
pilots  reported  noticing  that  the  heading  tape  had  been  frozen.  Freezing  the  instruments  may  be 
an  interesting  (albeit  limited)  way  to  index  SA  in  future  research.  It  is  limited  because  overuse 
of  this  approach  might  sensitize  the  pilots  to  potential  problems  with  the  instruments  and  cause 
even  more  cognitive  tunneling  on  the  HUD,  an  outcome  that  could  interfere  with  the 
interpretation  of  the  results. 

1.3.2  -  Post  Experiment  Comparison  of  HUD  to  HDD 

At  the  end  of  the  second  day  of  testing,  pilots  compared  the  HUD  and  HDD  conditions 
on  a  variety  of  measures,  similar  to  those  they  had  been  evaluating  throughout  the  experiment 
(See  Annex  B).  To  test  whether  the  pilots  perceived  a  difference  between  the  two  conditions, 
mean  ratings  were  compared  to  the  value  ‘4’  representing  “no  difference”  between  the  HUD  and 
HDD  conditions.  The  mean  values  and  standard  errors  are  shown  in  Table  1.7. 
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Table  1.7:  Comparison  between  HUD  and  HDD  Flight 

M 

SE 

t(3) 

Overall  situation  awareness 

4.3 

0.8 

0.29 

Awareness  of  heading 

3.5 

1.3 

-0.40 

Awareness  of  altitude 

4.0 

1.2 

0.00 

Awareness  of  airspeed 

3.5 

0.6 

-0.77 

Awareness  of  spatial  orientation 

3.5 

0.5 

-1.00 

Awareness  of  activity  on  the  ground 

4.3 

0.5 

0.52 

Awareness  of  activity  in  the  air 

4.5 

0.6 

0.77 

Awareness  of  aircraft  systems 

4.3 

1.5 

0.38 

Cross  checking  relevant  instruments 
/symbology 

4.3 

0.8 

0.29 

Using  information  from  the  scene  to  control 
the  aircraft 

3.8 

0.3 

-1.00 

Eyes-out  time 

1.8 

0.3 

-9.00** 

Low-level  flight 

2.8 

0.6 

-1.99 

Low-level  maneuvering 

3.0 

0.8 

-1.22 

Maintaining  airspeed 

4.0 

0.7 

0.00 

Maintaining  heading 

3.5 

0.6 

-0.77 

Maintaining  altitude 

4.5 

0.9 

0.58 

Note:  1  =  HUD  much  better;  4  =  no  difference;  7  = 

**  p  <  .01 

instruments  much  better 

All  four  pilots  rated  the  HUD  as  much  better  than  the  HDD  condition  for  increasing 
“eyes  out”  time.  Thus,  pilots  perceived  an  advantage  for  the  HUD  only  in  the  sense  that  they  felt 
they  were  more  likely  to  be  looking  out  of  the  cockpit.  As  is  evident  from  both  SA  performance 
and  from  ratings,  however,  the  HUD  was  not  better  for  gaining  flight  information  and  was  worse 
for  actually  noticing  objects  in  the  environment.  Although  pilots  reported  some  difficulties  and 
perceived  performance  decrements  with  the  HUD,  they  did  not  translate  this  into  an  overall  more 
negative  evaluation.  It  is  clear  that  the  combination  of  the  subjective  and  objective  measures  is 
important  for  understanding  the  effects  of  the  unfamiliar  HUD  symbology  on  workload,  situation 
awareness,  and  performance. 

1.4  Discussion 

The  results  of  this  experiment  support  the  validity  of  the  CSE  framework  for  evaluating 
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M&S  assessment  of  technology.  Pilots  reported  perfonnance  differences  with  unfamiliar  (HUD 
condition)  versus  familiar  (HDD  condition)  versions  of  the  instruments.  These  perceived 
performance  decrements  were  reflected  in  poorer  SA,  specifically  resulting  in  a  decrease  in 
pilots’  ability  to  report  the  presence  of  objects  such  as  fighter  jets  and  other  helicopters  in  the 
external  scene.  Pilots’  objective  workload  was  also  worse  when  using  the  unfamiliar  HUD 
symbology,  suggesting  that  they  experienced  either  a  high  level  of  workload  or  cognitive 
tunneling.  Furthermore,  pilots  showed  a  tendency  to  rate  their  overall  SA  as  slightly  better  in  the 
HDD  condition  than  with  the  HUD  condition,  but  did  not  perceive  differences  in  workload. 

Thus,  the  combination  of  measures  used  in  this  study  showed  that  although  patterns  of  situation 
awareness,  workload,  and  task-relevant  perfonnance  varied  across  the  HUD  and  HDD 
conditions,  the  objective  and  subjective  indices  of  these  constructs  were  not  always  perfectly 
aligned. 

The  overall  level  of  SA  shown  by  these  pilots  was  moderate  -  even  in  the  HDD 
condition,  they  missed  approximately  40%  of  the  objects  in  the  OTW  scene.  In  actual  flight,  of 
course,  the  pilot’s  eyes  are  augmented  by  those  of  the  other  crewmembers  and  their  performance 
occurs  as  part  of  a  team  effort.  Furthermore,  these  particular  pilots  had  no  direct  experience  with 
HUDs  and  the  two  aspects  of  the  task  that  they  found  most  difficult,  maintaining  speed  and 
altitude,  are  arguably  the  indices  that  are  most  different  across  the  HUD  and  HDD  conditions. 
The  difficulties  reported  by  the  pilots  may  reflect  interference  from  their  relatively  automatic  and 
well-trained  perception  of  the  HDD  values  versus  the  novel  HUD  values.  Furthermore,  indices 
of  both  airspeed  and  ground  speed  are  indicated  on  the  HUD  and  pilots  commented  that  they 
found  it  confusing  to  have  these  two  similar  values  placed  closely  together. 

The  novelty  of  the  format  for  speed  and  altitude  may  not  convey  the  whole  story, 
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however.  The  representation  of  the  heading  information  is  also  quite  different  in  the  HUD  as 
compared  to  the  HDD.  In  the  HDD  panel,  heading  appears  as  a  compass  (viewed  from  above). 
In  the  HUD,  heading  appears  as  a  continuous  tape,  with  the  pilot’s  current  heading  always 
displayed  directly  forward.  Despite  the  novelty  of  the  heading  information,  the  pilots  did  not 
report  finding  it  more  difficult  to  maintain  heading  or  to  get  information  from  the  heading  tape, 
but  tended  to  rate  it  as  similar  or  easier  to  use  than  the  compass  representation  of  heading  that 
they  were  familiar  with.  Importantly,  the  questionnaire  used  in  the  present  research  was 
sensitive  to  variations  in  the  relative  familiarity  and  useability  of  the  various  values. 
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Section  II:  Overview  of  the  Carleton  University  Simulator  Facility  and 

Progress  Report 


2.1  Overview 

The  flight  simulator  at  Carleton  University  is  a  custom  version  of  the  Networked 
Tactical  Simulator  (NTS)  that  has  been  developed  by  the  HFE  Group  as  part  of  the 
TAMSS  initiative.  As  with  other  NTS  systems,  the  Carleton  NTS  represents  the  flight 
deck,  mission  equipment,  and  physical  structure  of  the  Department  of  National  Defence’s 
(DND)  CHI 46  Griffon  helicopter.  The  Carleton  simulator  includes  both  out-the- window 
(OTW)  and  Helmet  Mounted  Display  (HMD)  capabilities.  The  Carleton  simulator  has  a 
head  tracker  to  support  the  use  of  the  HMD.  Additionally,  the  simulator  supports  the 
creation  of  synthetic  environments  and  scenario  creation  via  staging  software. 

The  Carleton  NTS  is  unique  in  that  it  includes  experimental  and  data  collection 
capabilities.  These  experimental  capabilities  allow  a  user  to  create  visual  and  auditory 
events  that  can  be  inserted  into  a  mission.  The  data  collection  capabilities  enable  an 
experimenter  to  examine  over  100  logged  measures  in  analyzing  the  perfonnance, 
workload,  and  situation  awareness  of  the  pilots  flying  the  simulator. 

The  Carleton  NTS  consists  of  six  PCs,  running  Windows  2000  Professional. 

Three  of  the  PCs  are  used  for  image  generation  (IG1,  IG2,  and  IG3)  and  simultaneously 
project  onto  three  8’  x  6’  screens,  providing  the  pilot  with  an  almost  180-degree 
horizontal  and  40-degree  vertical  view.  Two  PCs  (INSTR1,  INSTR2)  are  used  for 
simulation  of  the  flight  model  and  instrumentation.  INSTR1  is  responsible  for  running 
the  helicopter  flight  model  (HELISIM),  simulating  the  avionics,  and  for  driving  the  pilot 
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instrument  panel.  INSTR2  is  responsible  for  the  operation  of  the  CDUs.  As  well,  the 
INSTR1  PC  hosts  the  custom  data  collection  software  used  in  the  Carleton  experiments. 

Finally,  the  sixth  PC,  the  Experimenter  Operating  Station  (EOS),  is  responsible  for 
overall  system  control,  including  mission  loading  and  unloading.  As  well,  this  PC  hosts 
the  scenario  generation  software  and  a  Stealth  viewer.  The  simulator  includes  an  ASTi 
Digital  Audio  Communications  System  (DACS)  that  supports  simulation  of  cockpit  voice 
communication  as  well  as  voice  communication  between  the  pilot  and  console  operator. 

Data  transfer  between  the  various  modules  occurs  via  one  of  three  modes.  High  volume 
data,  such  as  that  between  the  flight  model  software  and  scene  generator,  uses  UDP 
communications.  Communications  between  the  avionics  simulation,  the  pilot  instrument 
panels  and  the  CDU  infrastructure  is  via  high-level  architecture  (HLA).  HLA  is  also 
used  to  interface  the  simulator  to  external  systems.  Finally,  shared  memory  is  used  for 
communications  between  the  CDU  and  the  CDU  Proxy.  Figures  2. 1  and  2.2  provide  a 
general  overview  of  the  hardware,  functionality,  and  communications  infrastructure  of 
the  Carleton  NTS. 

2.2  Hardware 

The  following  summarizes  the  main  components  of  the  Carleton  NTS.  The  six  PCS  in 
the  Carleton  simulator  are  equipped  with  dual  Pentium  III  1GHz  processors.  Each  machine  has  1 
GB  of  RAM.  The  machines  are  physically  networked  using  a  3Com  SuperStack  3  Baseline 
10/100  12  port  Ethernet  Switch.  Three  NEC  Model  MT1055  Data  Projectors  are  used  to  project 
generated  images  (using  VEGA  software)  on  8’x  6’  screens,  creating  the  immersive  out-the- 
window  scene.  The  headtracking  system  used  is  an  Intersense  IS-900  Virtual  Workbench 
Tracking  System.  The  IS-900  is  a  6  degrees-of-freedom  tracker,  tracking  both  position  and 
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angular  changes  (X,  Y,  Z,  Heading,  Pitch  and  Roll).  The  IS-900  provides  position  resolution  of 
1 .5mm  in  position  and  an  angular  resolution  of  0.05  degrees.  It  is  jitter-free  with  a  position 
stability  of  4  mm  and  angular  stability  of  0.2,  0.4  RMS.  IVision  Virtual  Reality  goggles  are  used 
as  the  ANVIS  Head-Up  Display  (HUD).  For  a  detailed  depiction  of  the  hardware  configuration 
of  the  Carleton  simulator,  refer  to  Figure  2.2. 

Figure  2.1  -  Carleton  Simulator  General  Overview 


D  ia  grain  provide  d  courtesy  of  the  HFE  Crroup 
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Figure  2.2  -  Carleton  Simulator  Hardware  Overview 


Diagram  provided  courtesy  of  the  HFE  Group 


2.3  Software  Overview 

2.3.1  COTS  Products 

The  simulator  software  consists  of  several  Commercial  Off-the-Shelf  Software  (COTS) 
packages  integrated  by  custom  control,  communications,  and  experimental  software.  The 
primary  COTS  products  used  and  their  function  within  the  Carleton  system  are  as  follows: 

VEGA  -  Vega,  a  product  from  Multigen-Paradigm  is  the  COTS  tool  used  to  render  the 
Out  the  Window  (OTW)  or  through  the  HMD  external  scene.  Vega’s  strength  is  that  it  is  able  to 
render  complex  geometries  in  real-time.  It  is  a  key  component  in  achieving  visual  realism  in  the 
simulation.  It  is  ultimately  based  on  the  OpenGL  2D  and  3D  graphics  application  programming 
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interface  (API). 

Vega  uses  Openflight  (a  3D  file  format)  models  of  both  the  terrain  database  and  scene 
objects  to  render  the  scene.  These  entities  are  configured  into  the  Vega  application  using  the 
LynX  graphical  interface.  This  graphical  interface  is  used  to  create  Vega  application  definition 
files  (.adf  files).  These  files  describe  both  graphical  and  platform  related  details  of  the  Vega 
application.  Vega  renders  the  outside  scene  based  on  the  graphical  objects  defined  in  the  .adf 
files  and  a  given  “eye”  point  detennined  by  the  aircraft  position  and/or  head  position. 

Vega  also  includes  a  development  API  that  enables  a  user  to  customize  Vega 
functionality  for  specific  applications.  For  example,  this  API  is  used  in  the  Carleton  system  to 
generate  the  HUD  information  and  symbology.  Vega  software  callbacks  are  used  to 
superimpose  the  HUD  information  on  the  Vega  scene  as  seen  through  the  goggles.  The  API  has 
also  been  used  to  extend  Vega  capabilities  to  handle  visual  experiment  events  generated  through 
the  experimental  software. 

HELISIM  -  Helisim,  from  Virtual  Prototypes  Incorporated,  is  a  software  package  used  to 
provide  the  flight  model.  HELISIM  mimics  the  perfonnance  of  a  rotary-wing  aircraft  by  tuning 
parameters  such  as  weight  and  balance,  propulsion  and  rotor  characteristics,  and  instrumentation, 
thus  enabling  the  simulator  to  closely  represent  the  flight  dynamics  of  the  CHI  46  Griffon. 
HELISIM  accepts  inputs  from  the  collective,  cyclic  and  pedals  of  the  simulator  and  using  the 
defined  flight  model,  updates  aircraft  position  (i.e.,  latitude,  longitude,  and  altitude),  aircraft 
heading,  pitch,  and  roll  as  well  as  several  other  flight  and  instrumentation  values. 

An  important  feature  of  HELISIM  is  an  API  that  allows  for  real-time  control  of  various 
aircraft  parameters.  This  is  an  extremely  important  feature  for  experimentation.  Using  these 
HELISIM  features,  the  Carleton  lab  has  developed  a  capability  to  freeze  specific  instrumentation 
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(e.g.,  aircraft  heading,  radar  altimeter,  etc.).  Pilots'  situation  awareness  of  their  cockpit  systems 
is  measured  using  the  freezing  capability  (e.g.,  did  the  pilot  notice  the  frozen  instrumentation, 
how  long  did  it  take  for  them  to  notice  and  react  to  the  frozen  instrumentation).  The  Carleton  lab 
has  also  used  the  HELISIM  API  to  develop  a  capability  to  measure  the  control  of  aircraft 
position  and  orientation  based  on  an  ADS-33  attitude  recovery  task. 

STAGE  -  STAGE  is  the  acronym  for  Scenario,  Toolkit  and  Generation  Environment.  It  is 
a  software  tool  used  to  create  complex  tactical  scenarios.  STAGE  provides  a  graphical  user 
interface  in  which  to  enter  information  into  a  tactical  database.  This  database  then  generates  the 
real-time  tactical  scenario.  STAGE  also  displays  the  real-time  positions  of  entities  in  the 
scenario  as  it  is  run  on  its  situation  display. 

STAGE  is  used  to  add  “entities”  to  the  simulated  mission  scenarios.  This  STAGE  entity 
information  is  sent  to  Vega,  which  renders  the  STAGE  entities  in  the  external  scene  in  the 
appropriate  position.  The  level  at  which  the  pilot  detects  the  STAGE  entities  during  the  mission 
can  be  used  to  gauge  the  pilot’s  level  of  situation  awareness. 

STAGE  can  be  run  in  one  of  two  modes  -  with  HLA  enabled  or  disabled.  When  HLA  is 
enabled,  STAGE  becomes  the  HLA  gateway  for  the  entire  system  and  can  be  used  to  send  the 
STAGE  entity  (including  Ownship)  infonnation  to  external  agencies.  When  STAGE’S  HLA  is 
not  enabled,  STAGE  communicates  only  with  the  other  simulator  components. 

STEALTH  -  The  MAK  Stealth  viewer  is  a  3D  visualization  tool  that  extends  the  console 
operator's  viewpoint  of  the  simulated  environment  beyond  the  fixed  point  of  the  pilot  to 
anywhere  in  the  simulated  world.  Stealth  enables  the  console  operator  to  attach  to  other  entities 
in  the  simulated  environment  to  see  the  world  through  their  eyes.  Stealth  receives  its  information 
on  entity  position  from  STAGE  using  the  HLA  interface. 


Centre  for  Applied  Cognitive  Research 


25 


TAMSS  SA  Experiment  1 


2.3.2  Custom  Code 

Custom  code  within  the  Carleton  simulator  is  used  for  the  Experimental  Operator  Station 
(EOS).  EOS  is  responsible  for  the  command  and  control  of  various  components  as  well  as  for 
the  unique-to-Carleton  experimental  and  data  logging  capabilities.  A  more  detailed  description 
of  the  EOS  capabilities  is  given  in  the  next  section.  Custom  code  is  also  used  to  develop  the 
HMD  symbology  generation  capabilities. 

Other  components  that  use  custom  code  are  the  avionics  simulation  module,  the  pilot 
Head-Down  Display  (HDD)  instrument  panel,  the  CDU  Bezel  and  CDUs,  and  the  HUD 
symbology.  The  Communications  software  within  the  simulator,  whether  via  HLA,  UDP  or 
Shared  Memory  is  also  custom  code.  Tools  used  in  the  development  of  the  custom  code  include 
VAPS  by  Virtual  Prototypes  Incorporated  (CDU),  GLStudio  (pilot  head-down  instrument  panel 
and  HUD  symbology)  by  DiSTI,  and  VR-Link  (HLA)  by  MAK  Technologies. 

2.4  The  Experimental  Operator  Station  (EOS) 

The  EOS  encapsulates  control  functions  for  the  Carleton  simulator  as  well  as  the 
experimental  capabilities  unique  to  the  Carleton  system. 

2.4.1  -  Command  and  Control  Capabilities  of  the  EOS 

The  control  functions  of  the  simulator  are  accessed  via  a  GUI  interface.  The  main  control 
functions  are  as  follows: 

1.  The  Mission  Control  function  opens,  loads,  resets,  unloads  and  exits  missions. 

2.  The  Location  function  allows  for  console  control  of  the  aircraft  position.  It  has  a  slew 
mode  in  which  the  operator  can  move  the  aircraft  in  all  cardinal  directions  as  well  as 
change  the  aircrafts  altitude  and  heading. 

3.  The  Weather  function  changes  atmospheric  conditions  such  as  wind  direction,  wind 
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speed  and  cloud  cover  in  the  OTW  scene. 

4.  The  Ownship  function  can  be  used  to  alter  the  ownship's  fuel  levels,  communications  and 
navigation  settings. 

5.  The  Options  function  currently  enables  the  operator  to  change  the  date  and  time  at  which 
the  mission  is  taking  place.  The  Vega  generated  scene  will  accordingly  adjust  lighting 
when  these  are  changed. 

6.  The  Communications  function  enables  the  operator  to  monitor  and  transmit  on  a  specified 
frequency  via  the  Digital  Audio  Communication  System  (DACS).  It  also  has  an  intercom 
capability. 

7.  The  Freeze  function  allows  the  operator  to  freeze  the  aircraft,  the  scenario  or  both  (freeze 
all). 

8.  The  System  Control  function  gives  the  operator  some  degree  of  remote  control  over  the 
PCs  in  the  network.  It  supports  global  Reboot,  Reboot  IGs,  and  global  Shutdown 
capabilities. 

2.4.2  -  Experimental  Capabilities  of  the  EOS. 

The  experimental  capabilities  of  the  Carleton  simulator  make  it  different  from  the  average 
flight  simulator.  The  Carleton  system  has  the  ability  to  generate  events,  log  pilots'  responses  to 
events,  and  log  key  flight-related  data  throughout  a  mission.  This  data  can  then  be  used  to  assess 
pilot  performance,  workload  and  situation  awareness. 

The  experimental  software  has  three  major  components:  a)  Experiment  Scenario  Generation, 
b)  Experiment  Control,  and  c)  Data  Collection.  The  interplay  of  these  three  components  is 
shown  in  Figure  2.3. 
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a)  Experiment  Scenario  Generation  -  The  Experiment  Scenario  Generation  creates  an 
experimental  scenario  by  defining  a  list  of  events  that  are  to  occur  during  the  course  of  the 
mission.  The  system  currently  supports  the  generation  of  audio  and  visual  events.  Audio  events 
generate  a  tone  of  specified  frequency  and  duration  using  the  DACS.  Visual  events  generate  an 
image  that  is  displayed  in  Vega  for  a  specified  duration  of  time.  The  experimenter  can  set  the 
onset  time,  timing  mode  (periodic  or  on-shot  events),  as  well  as  the  duration  of  the  probe.  For 
periodic  timing,  the  experimenter  specifies  the  period  interval  as  well  as  the  variance  in  interval 
time.  For  visual  events,  the  experimenter  can  also  specify  the  reference  point  (e.g.,  Head, 
Aircraft,  or  World  reference),  as  well  as  the  relative  position  to  the  frame  center  (e.g.,  X,  Y,  Z, 
Heading,  Pitch,  Roll)  at  which  the  event  should  appear. 

Figure  2.3  -  Experimental  System  Overview 
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Diagram  provided  courtesy  of  the  HFE  Group 
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There  are  two  types  of  tasks  associated  with  the  event,  detection  and  discrimination, 
which  are  currently  supported  by  the  system.  In  the  detection  task,  the  pilot’s  task  is  to  detect 
audio  or  visual  stimuli  and  respond  as  quickly  as  possible  to  the  stimuli,  usually  by  pressing  a 
button.  In  the  discrimination  task,  the  pilot's  goal  is  to  identify  a  specific  object  out  of  a  range  of 
objects,  using  a  TRUE/FALSE  type  of  response.  The  user  is  able  to  specify  the  number  of 
objects  to  be  used  in  the  experiment  as  well  as  the  weighting  of  TRUE  to  FALSE  responses. 

Once  all  the  events  have  been  specified,  the  experimental  scenario  is  saved  as  an  “.exp” 
file,  and  the  experiment  is  ready  to  be  launched. 

b)  Experiment  Control  -  Experiment  Control  launches  the  experimental  scenario  and 
is  used  to  initiate  the  data  collection  process  once  the  desired  “.exp”  file  has  been  selected  by 
the  experimenter.  A  graphical  interface  enables  the  user  to  select,  launch  and  stop  an 
experimental  scenario.  The  EOS  feeds  information  from  the  “.exp”  file  to  an  event  scheduler 
which  initiates  the  audio  and  visual  events  at  the  designated  times.  Experiment  Control  also 
contains  a  head  tracker  calibration  utility. 

c)  Data  Collection  -  Data  Collection  is  a  background  process  that  starts  automatically 
when  the  experiment  is  launched.  The  Data  collection  application  logs  the  following 
information: 

1 .  Session  infonnation:  Session  ID,  Start  Time,  End  Time,  Subject  Number,  Trial  Number 

2.  Audio  Events:  Session  ID,  Simulator  Time  (of  event),  Task  Type,  Event  Duration,  Truth 
Value,  IsTarget  (in  discrimination  task),  Tone  Frequency 

3.  Visual  Events:  Session  ID,  Simulator  Time  (of  event),  Task  Type,  Event  Duration,  Truth 
Value,  Is  Target  (in  discrimination  task),  Object  Number,  Reference  (Head,  Aircraft, 
World),  X,  Y,  Z,  H,  P,  R 
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4.  Button  Press:  Session  ID,  Simulator  Time  (of  button  press),  Button  Number  Pressed 

5.  Headtracker  Data:  X,  Y,  Z,  H,  P,  R  (Positional  and  Angular  information) 

6.  HELISIM  Data:  Session  ID,  Simulator  Time  (of  data)  plus  99  other  parameters  derived 
by  HELISIM.  The  most  important  of  these  for  experimental  purposes  are:  Latitude, 
Longitude,  Altitude,  Speed  and  Heading. 

The  above  items  are  logged  in  separate  SQL  tables  and  can  be  examined  and  extracted  using 
a  Microsoft  Access  application  as  a  front-end  to  the  database. 

Because  the  Carleton  simulator  is  used  for  experimentation,  it  is  important  to  determine  the 
amount  of  time  delay  occurring  between  a  pilot's  action  or  response  in  the  simulator  and  the 
logging  of  this  response  by  the  Experimental  software.  Ideally  this  delay  will  be  minimal  so  that 
the  simulator  mimics  real-time  events  as  closely  as  possible.  Lurthermore,  the  amount  of  time 
that  it  takes  for  information  to  pass  through  the  system  to  the  EOS  should  be  consistent  over  time 
to  ensure  that  the  measures  being  sampled  are  accurate.  The  next  section  will  examine  the  tests 
that  have  been  performed  by  the  CACR  in  an  attempt  to  measure  the  time  delays  occurring 
within  the  Carleton  simulator. 

2.4.3  Experimental  features  added 

In  addition  to  the  experimental  features  provided  by  the  HLE  group,  the  CACR  has 
successfully  added  several  new  experimental  features  to  the  simulator.  These  include  add-ons 
and  upgrades  to  the  HFE  Group’s  original  applications,  as  well  as  new  applications  allowing  for 
new  experimental  designs  and  measurement.  This  section  will  briefly  review  these  features. 

Freezing  of  the  HUD  -  This  feature  allows  the  operator  of  the  experiment  to  freeze  at  any 
given  moment  any  of  the  following  indications  displayed  on  the  HUD: 

•  Air  speed 
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•  Heading  tape 

•  Altitude 

This  feature  is  useful  for  Situation  Awareness  (SA)  experiments  as  the  operator  can  record  the 
amount  of  time  that  it  takes  for  a  pilot  to  notice  the  frozen  indicators.  This  feature  also  records 
the  time  at  which  the  freeze  command  was  initiated,  allowing  the  experimenters  to  refer  to  this 
information  at  a  later  time.  This  feature  also  provides  the  operator  with  the  ability  to  remove  and 
replace  the  entire  HUD  display  upon  command. 

Freeze  HDD  Instruments  -  An  application  has  been  developed  to  give  the  operator  of  the 
experiment  the  ability  to  freeze  at  any  given  moment  any  of  the  following  indications  displayed 
on  the  HDD: 

•  Air  speed 

•  Heading 

•  Radar  Altitude 

•  Barometric  Altitude 

•  Torque 

This  feature  is  useful  for  Situation  Awareness  (SA)  experiments  as  the  operator  can  indicate 
when  or  if  the  pilot  noticed  that  the  indicator  was  frozen.  This  application  also  records  the 
time  at  which  the  freeze  command  was  initiated. 

Attitude  recovery  task  -  Although  not  used  during  Experiment  1,  a  complete  attitude 
recovery  task  was  created.  The  task  consists  of  blanking  the  screen,  placing  the  aircraft  at  a 
pre-con ligured  attitude,  then  un-blanking  the  screen  and  passing  control  to  the  pilot.  The 
pilot  is  then  instructed  to  indicate  their  completion  of  recovery  by  pressing  one  of  the  cyclic 
buttons.  During  the  recovery  process,  numerous  data  are  recorded  (in  addition  to  aircraft 
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attitude)  including  the  time  at  which  the  scene  was  un-blanked  and  control  was  passed  to 

pilot  and  the  time  at  which  the  pilot  indicated  task  completion. 

This  application  also  allows  for  the  task  difficulty  to  be  controlled  by  the  experimental 
operator.  This  control  can  be  achieved  by  controlling/configuring  the  initial  aircraft  attitude  and 
by  controlling/configuring  the  turbulence  at  the  area  where  the  task  takes  place.  In  addition,  the 
following  task  parameters  can  be  configured  by  the  experimental  operator  through  the  use  of  a 
task  configuration  table  (a  simple  text  file): 

1 .  Air  speed 

2.  Altitude 

3.  Heading 

4.  Pitch 

5.  Roll 

6.  Torque  -  This  is  optional  in  cases  when  inexperienced  pilots  are  being  used. 

7.  Turbulence  -  vertical  air  speed,  horizontal  air  speed,  period  of  cycles. 

Cyclic  button  press  recording  and  filtering  -  A  simple  filtering  mechanism  has  been  added  to 
clear  the  noise  that  is  recorded  when  a  pilot  presses  and  holds  down  the  cyclic  button  for  a  long 
period  of  time.  A  thread  was  added  to  allow  for  a  cyclic  button  press  sampling  rate  of  5  ms.  This 
sampling  rate  can  be  adjusted  as  necessary. 

2.4.4  General  additions  and  upgrades 

A  final  addition  to  the  Carleton  simulator  system  is  an  interface  for  HELISIM  that  allows 
developers  to  fly  the  aircraft  from  the  development  station  using  a  simple  joystick.  This 
interface  was  developed  to  enable  the  developer  to  fly  the  aircraft  while  using/monitoring  other 
tools  at  the  development  station. 
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The  addition  of  these  experimental  features  will  allow  the  CACR  team  to  measure  the  SA 
of  pilots  under  several  new  conditions  (i.e.,  when  the  HUD  or  HDD  freezes,  or  when  they  must 
recover  after  losing  control  of  the  aircraft).  These  measures  will  assist  in  the  definition  and 
measurement  of  SA. 

2.5  Carleton  Simulator  System  Timing  -  Evaluation  and  Testing 

2.5.1  UDP  Packet  Delay  Measures 

The  CACR  engineers  have  taken  measurements  to  determine  the  amount  of  time  required  to 
transfer  a  single  UDP/IP  packet  between  two  applications  running  on  two  separate  PCs.  UDP/IP 
packets  are  the  main  data  transfer  mechanism  used  by  the  simulator.  They  are  used  both  for 
direct  communication  between  the  several  modules  that  make  up  the  simulator  system  and  as  a 
transport  layer  for  “over  HLA”  communication.  The  delay  of  a  single  small  UDP  packet  sent 
between  two  applications  running  on  two  separate  machines  was  measured  by  the  CACR.  This 
measurement  was  gained  using  loop-back  methodology  and  was  based  on  the  internal  Windows 
clock  giving  a  resolution  of  1ms.  It  was  found  that  the  measured  delay  in  a  loaded  system  at  a 
steady  state  (while  executing  standard  Out  the  Window  scene)  is  always  less  than  2  ms. 

2.5.2  Serial  Port  Loop-back  delay 

The  delay  between  the  writing  and  the  reading  of  a  single  ASCII  character  through  the  PCs 
serial  port  was  measured  by  means  of  an  external  loop-back.  When  a  single  ASCII  character  is 
written  to  one  of  the  PC's  serial  ports,  it  triggers  the  digital  scope.  The  digital  scope  is  used  for 
several  timing/delay  measurements  in  a  system. 

In  order  to  perform  this  measurement,  the  Tx  pin  of  the  serial  port  was  shortened  externally 
with  the  Rx  pin.  Then,  a  Windows  application  was  used  to  send  a  single  character  (8  bit)  at 
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115600  bits/sec  using  the  standard  Windows  WriteFile()  call  and  receive  the  character  using  the 
standard  Windows  blocking  ReadFile()  call.  The  delay  between  the  two  calls  was  measured 
using  the  internal  Windows  clock  with  a  resolution  of  1ms.  The  measured  delay  between  the  two 
calls  was  found  to  be  less  than  2  ms.  This  measurement  gives  us  an  indication  of  the  interaction 
delays  in  our  system  between  the  application  layer  and  the  hardware  layer.  These  interactions 
are  carried  out  by  the  Windows  device  drivers. 

2.5.3  Vega  timing  (for  post-draw  drawing ) 

This  measurement  was  taken  to  determine  the  amount  of  delay  that  occurs  between  the 
time  that  an  object  is  added  to  the  post-draw  function  in  Vega  and  the  time  at  which  the  object 
appears  on  the  projection  screen.  The  last  stage  at  which  Vega  can  execute  a  user  code  before 
the  display  buffers  for  the  current  frame  are  swapped  and  then  displayed,  is  during  the  post-draw 
callback  stage.  The  user  can  add  visual  objects  such  as  the  HUD  symbology  or  any  other  visual 
objects  at  this  stage.  The  CACR  team  measured  this  delay  using  the  following  process: 

1 .  A  method  that  blanked  the  scene  (the  display  buffer)  was  called. 

2.  A  method  to  draw  a  small  bright  (white)  rectangle  at  the  top  of  the  scene  was  called. 

3.  A  signal  to  trigger  a  two-channel  digital  scope  was  sent  through  the  serial  port  to  the 
scope. 

4.  Control  was  taken  by  Vega  that  swapped  the  display  buffer. 

This  process  allowed  transmission  the  photo  diode  that  was  connected  to  the  second  channel  of 
the  digital  scope  through  an  amplifier  to  send  a  signal  to  the  scope  as  soon  as  the  bright  rectangle 
appeared  at  the  top  of  the  screen.  The  delay  between  the  time  that  an  object  was  added  as  a  post¬ 
draw  object  and  the  time  that  it  took  to  appear  at  the  top  of  the  screen  (in  this  case  as  a  white 
rectangle)  was  28  ms  (±lms).  When  the  same  bright  rectangle  was  moved  to  the  bottom  of  the 
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screen,  the  measured  delay  was  44  ms  (±lms)  that  is,  28  ms  +  1/60  Hz.  It  is  important  to  note 
that  the  delay  between  the  end  of  the  post-draw  stage  (that  is  under  the  user’s  code  control)  and 
the  time  when  the  actual  image  starts  to  appear  at  the  top  of  the  screen  should  remain  constant 
regardless  of  the  amount  of  displayed  data. 

2.5.4  Cyclic  button  press  delay 

The  latency  between  a  button  press  on  the  cyclic  and  the  time  it  takes  to  receive  this  response 
at  the  PCI  based  A/D  board  was  measured.  In  the  SA  Experiment  1,  a  button  press  on  the  cyclic 
was  used  by  the  pilots  in  response  to  stimuli  that  were  presented  as  part  of  the  scenario. 

Originally  the  pilots'  controls  (including  the  cyclic  buttons)  were  sampled  by  HELISIM  at  a  rate 
of  60  Hz,  meaning  there  was  a  minimum  delay  of  16.66  ms.  A  separate  thread  has  been  added  by 
the  CACR  to  make  the  sampling  rate  independent  of  HELISIM.  This  has  allowed  for  a  sampling 
rate  much  faster  than  60Hz.  In  future  applications,  the  delay  will  be  re-measured  with  the 
addition  of  this  new  thread.  Sending  a  character  through  one  of  the  serial  Tx  pins  and  connecting 
it  externally  to  one  of  the  discrete  inputs  on  the  A/D  board  will  gain  this  measure.  The  time 
difference  between  the  call  to  WriteFile()  and  the  time  that  a  change  in  the  state  of  the  discrete 
input  is  sensed  by  the  thread  will  be  measured  using  the  Windows  clock. 

2.5.5  Time  Synchronization  mechanism 

A  time  synchronization  mechanism  to  synchronize  the  6  PCs  in  the  simulator  has  been 
implemented.  This  mechanism  consists  of  a  timeserver  that  executes  on  one  of  the  PCs  and  time 
clients  that  periodically  (at  ten  second  intervals)  send  time  synchronization  requests  to  the  server. 
These  time  clients  also  receive  updated  time  responses  that  are  used  to  set  their  PC's  internal 
clock. 
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As  part  of  this  process,  half  of  the  travel  delay  (the  time  between  sending  the  sync  request 
and  receiving  the  time  sync  replay)  is  subtracted  at  the  client  side  to  compensate  for  the 
communication  delay  between  the  client  and  the  server  (if  the  measured  round  trip  delay  is 
longer  than  4  ms  the  clock  is  not  getting  updated  during  that  cycle).  A  UDP/IP  packet  with  a 
delay  of  less  then  2  ms  was  sent  between  the  PCs  to  determine  the  degree  of  synchronicity 
between  the  PC  clocks.  The  clock  synchronization  in  the  system  was  found  to  be  ±2ms.  This 
mechanism  can  be  used  to  obtain  accurate  response  measurements  in  the  system,  especially  when 
an  event  is  initiated  by  an  entity  executing  on  one  machine  while  the  response  for  the  event  is 
accepted  by  an  entity  executing  on  a  different  machine. 
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Section  III:  Conclusions 

3.1  Cognitive  Systems  Engineering  (CSE)  Framework 

Recent  advances  in  both  knowledge  of  human  psychology  and  the  capabilities  of 
simulation  environments  support  a  greater  role  for  modeling  and  simulation  (i.e.,  M&S)  in 
human  factors  research.  The  CACR  Cognitive  Systems  Engineering  (CSE)  framework  has  been 
proposed  as  a  guide  for  the  evaluation  component  of  the  M&S  process.  The  goal  of  the  CSE 
framework  is  to  provide  general  guidelines  for  evaluating  new  technologies  from  the  perspective 
of  how  these  will  affect  the  human-machine  interface. 

The  results  of  the  experimental  research  that  was  conducted  in  the  present  report  support 
the  CSE  framework.  In  particular,  this  research  shows  that  converging  measures  of  performance, 
workload  and  SA  should  be  obtained  in  modelling  and  simulation  studies,  thereby  allowing 
researchers  to  obtain  an  overall  perspective  that  does  not  rely  solely  on  a  single  construct  or 
single  method  of  measurement. 

3.2  Carleton  Simulator  Environment 

The  experimental  research  reported  in  this  document  demonstrated  that  the  Carleton 
simulator  facility  is  an  effective  environment  for  conducting  experimental  tests  of  M&S  efforts. 
The  development  tools  allowed  for  the  creation  and  implementation  of  appropriate  mission 
scenarios  (through  the  use  of  STAGE)  and  for  the  use  of  visual  and  auditory  probes  to 
objectively  measure  workload.  The  simulator  communications  utility  was  effective  in  allowing 
the  experimenters  to  interact  with  the  pilot.  In  addition,  the  experimenters  were  able  to  maintain 
awareness  throughout  each  mission  in  the  study  through  the  use  of  the  experimenter-station 
displays.  Importantly,  the  pilots  who  participated  in  this  experiment  were  quickly  able  to  reach 
an  acceptable  level  of  comfort  in  flying  the  simulator.  The  fidelity  of  the  simulator  image 
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generation,  flight  controls,  communications  system  and  the  simulated  avionics  package  was 
acceptable  to  the  pilots. 

The  fundamental  structure  of  the  data  collection  and  logging  procedures  were  proven  to 
be  generally  acceptable.  However,  further  developments  of  the  Carleton  simulator  are  required 
to  more  fully  automate  data  collection  (including  the  recording  of  objects  in  the  environment). 
In  addition,  detailed  post-experiment  analysis  has  revealed  that  the  simulator  produces  some 
inconsistency  in  data  logging  that  needs  to  be  corrected  and  verified. 

Operation  of  the  simulator  was  found  to  be  less  stable  than  desired,  resulting  in 
occasional  (and  unpredictable)  system  crashes.  Efforts  are  ongoing  to  provide  for  a  more  stable 
system.  System  stability  will  become  increasingly  critical  when  connecting  to  other 
models/simulators  via  HLA. 
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Cognitive  Systems  Engineering  Framework 
for  Modeling  and  Simulation  in  the  Acquisition  Process 


Summary 


Military  aircraft  provide  a  forum  for  the  development  of  new  aviation  technology  that  has 
potentially  wide  application.  For  example,  heads-up  displays,  a  technology  that  is  common  in 
military  contexts,  are  just  becoming  widespread  in  commercial  aircraft.  In  the  past,  the 
implications  of  human-machine  interactions  were  rarely  explored  thoroughly  when  new 
technology  was  added  to  existing  systems.  With  the  development  of  affordable  simulation 
environments,  however,  the  feasibility  of  testing  new  technologies  before  they  are  installed  in 
aircraft  has  increased  substantially.  Savings  in  terms  of  human  costs  and  technology  retrofits  are 
potentially  enormous.  Furthermore,  there  have  been  recent  advances  in  both  our  knowledge  of 
human  psychology  and  the  capabilities  of  simulation  environments,  supporting  a  greater  role  for 
modeling  and  simulation  (i.e.,  M&S).  In  this  report,  we  propose  a  cognitive  systems  engineering 
(CSE)  framework  to  guide  the  evaluation  component  of  the  M&S  process.  The  goal  of  the  CSE 
framework  is  to  provide  general  guidelines  for  evaluating  new  technologies  from  the  perspective 
of  how  these  will  affect  the  human-machine  interface.  This  initial  report  consists  of  three  parts. 
Section  I  is  an  overview  of  the  proposed  CSE  framework.  Section  II  is  a  more  detailed  literature 
review  upon  which  the  framework  was  based.  Section  III  is  a  comprehensive  bibliography. 
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Section  I:  Proposed  Cognitive  Systems  Engineering  Framework 
I. A.  Overview  of  the  Proposed  Framework 

The  proposed  framework  for  evaluating  new  technologies  in  the  M&S  process  is  based 
on  the  application  of  a  human  factors  approach  to  understanding  cockpit  design.  In  the 
Cognitive  Systems  Engineering  (CSE)  framework,  we  assume  that  the  limitations  of  the  human, 
rather  than  of  the  technology,  must  guide  the  development  of  new  aviation  systems.  Pilots  are 
extremely  highly  skilled  operators  of  extremely  complex  machines.  Improvements  in  technology 
must  be  designed  around  an  understanding  of  the  strengths  and  limitations  of  the  human 
operator.  Thus,  the  proposed  CSE  framework  incorporates  understanding  of  human  cognition  in 
a  working  blueprint  for  the  design  of  evaluation  experiments  to  support  modeling  and  simulation 
in  acquisition.  As  shown  in  Figure  1.1,  the  framework  includes  a  theoretical  construct,  the 
dynamic  mental  model,  and  three  empirical  (i.e.,  measurable)  constructs;  situation  awareness, 
workload,  and  task-relevant  perfonnance.  Our  review  of  the  literature  indicated  that  these  three 
empirical  constructs  capture  a  large  amount  of  the  variance  in  the  human-machine  interface.  The 
proximal  goal  of  the  CSE  framework  is  to  provide  a  context  within  which  to  interpret  the 
dependent  variables  that  are  assessed  in  an  M&S  evaluation  (i.e.,  how  to  assess  the  benefits  of  a 
new  technology  in  a  complex  environment  such  as  the  Griffon  CH146  helicopter).  It  is  not 
intended  to  represent  a  complete  model  either  of  the  human  operator  or  of  the  situation,  although 
further  developments  of  the  framework  will  be  designed  to  expand  the  theoretical  and  predictive 
power  of  the  model. 

The  central  theoretical  construct  in  the  CSE  framework  is  the  dynamic  mental  model.  It 
captures  the  notion  that  the  human  operator  constantly  creates  and  maintains  an  internal 
representation  of  the  ongoing  situation.  When  experimental  methods  are  used  to  measure 
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performance  in  an  M&S  evaluation,  all  of  the  measurements  are  inferences  about  the  operator’s 
dynamic  mental  model.  The  three  empirical  constructs,  situation  awareness,  workload,  and  task¬ 
relevant  performance,  are  assumed  to  provide  a  comprehensive  (although  not  exhaustive) 
assessment  of  the  dynamic  mental  model.  Situation  awareness,  defined  simply  as  “knowing 
what  is  going  on  around  you”  (Endsley,  2000,  p.  5)  is  a  construct  that  was  proposed  originally  to 
capture  why  some  fighter  pilots  were  more  successful  (and  therefore  lived  longer)  than  others 
(Spick,  1988).  As  discussed  by  Endsley  (2000),  SA  is  closely  tied  to  knowing  how  to  distinguish 
important  infonnation  in  the  environment  from  less  important  infonnation  (selective  attention), 
as  well  as  the  ability  to  quickly  comprehend  the  importance  of  changes  to  elements  in  the 
environment. 

The  second  empirical  construct  that  forms  the  core  of  the  CSE  framework  is  workload. 
Workload  is  a  familiar  construct  in  aviation  and  has  been  studied  more  thoroughly  than  SA. 
Essentially,  workload  refers  to  the  fact  that  humans  are  limited  in  their  ability  to  process 
information  and  to  respond  appropriately.  A  helicopter  pilot  on  a  SAR  mission  who  is  flying 
with  night  vision  goggles  near  the  ground  during  a  rain  stonn  is  likely  to  be  in  a  situation  of 
heavy  workload.  A  large  amount  of  information  that  changes  rapidly  has  to  be  monitored  and 
the  pilot  must  constantly  be  updating  his  or  her  dynamic  mental  model  of  the  environment.  In 
contrast,  a  fighter  pilot  flying  in  good  weather  on  a  routine  recce  is  probably  in  a  low  workload 
situation.  Workload  has  proven  to  be  a  very  useful  construct  for  understanding  changes  in  pilots’ 
behaviour  under  different  situational  demands  and  constraints.  Technology  “improvements” 
should  hypothetically  decrease  workload,  but  in  practice,  a  technology  that  adds  infonnation  to 
the  pilot’s  environment  and/or  requires  the  pilot  to  perform  additional  tasks  is  more  likely  to 
increase  workload.  Thus,  measuring  changes  in  workload  as  a  function  of  technology  changes  is 
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crucial  to  understanding  how  that  technology  influences  pilots’  performance. 

The  third  empirical  construct  in  the  framework,  task-relevant  performance,  refers  to  the 
actions  of  the  pilot  (in  relation  to  mission  demands)  that  are  potentially  affected  by  the  new 
technology.  In  essence,  a  new  technology  is  expected  to  change  some  aspect  of  what  the  pilot 
knows  and  that  knowledge  will  be  reflected  in  his  or  her  behaviour.  Many  other  aspects  of 
pilots’  actions  or  behaviour  might  not  change,  however.  The  task-relevant  performance  that  is 
relevant  to  any  particular  technological  change  will  depend  on  what  that  technology  was 
expected  to  influence.  For  example,  the  addition  of  a  new  display  screen  to  the  F-18  cockpit  that 
has  enhanced  information  about  approaching  threats  should  result  in  the  pilot  spending  time 
looking  at  that  screen  and  interacting  with  it  in  certain  ways.  Concomitantly,  the  F-18  pilot  may 
spend  less  time  using  other  sources  of  information,  or  may  use  that  information  differently. 

Thus,  defining  and  measuring  task-relevant  perfonnance  is  an  important  aspect  of  understanding 
the  impact  of  a  new  technology  on  performance  in  the  cockpit. 
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FIGURE  1.1  -  CSE  FRAMEWORK 


SITUATION  AWARENESS 
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In  the  CSE  framework,  the  three  experimental  constructs  (SA,  workload,  task-relevant 
performance)  are  second-order  reflections  of  the  pilot’s  dynamic  mental  model  of  the  situation. 
Because  it  is  impossible  in  practice  to  directly  measure  the  contents  of  a  human  brain,  use  of 
multiple  measures  of  the  dynamic  mental  model  is  likely  to  provide  more  useful  information  than 
focusing  on  a  single  construct  such  as  situation  awareness.  Furthermore,  information  acquired 
through  the  research  process  is  limited  by  the  types  of  measures  chosen  and  by  the  underlying 
theoretical  assumptions  that  guided  the  choice  of  those  measures.  In  Section  II,  we  present  a 
detailed  overview  of  the  strengths  and  weaknesses  of  the  constructs  of  situation  awareness  and 
workload.  The  bottom-line  conclusion  is  that  any  single  measure  is  unlikely  to  provide  sufficient 
information  to  allow  for  general  conclusions  about  the  impact  of  a  new  technology  on  the  crew. 

Because  it  is  impossible  to  directly  measure  the  dynamic  internal  model  (as  it  is  not 
possible  to  directly  measure  “memory”  or  “thinking”),  all  measures  we  can  use  are  behavioural 
in  the  sense  that  the  pilot  or  crew  must  perform  some  action  that  is  then  assessed.  For  example,  a 
subjective  assessment  of  situation  awareness  requires  that  the  pilot  make  a  judgment  or  provide 
an  evaluation.  Head  position  could  be  used  as  an  index  of  where  the  pilot  is  attending.  Altitude 
maintenance  could  be  used  as  an  index  of  the  pilot’s  adherence  to  the  flight  plan.  In  the  case  of 
SA,  the  simulation  can  be  frozen  and  then  pilots  can  be  asked  to  report  instrument  information 
(such  as  altitude),  or  to  make  a  prediction  about  the  trajectory  of  enemy  planes  (i.e.,  bogies). 
Their  answers  to  such  queries  are  assumed  to  reflect  their  interrogation  of  their  dynamic  mental 
model  (i.e.,  memory  for  the  current  location  of  bogies  and  knowledge  about  what  those  bogies 
are  likely  to  do).  In  general,  the  researcher  assumes  that  the  behavioural  responses  or  actions  can 
be  used  as  an  index  of  the  pilot’s  knowledge  or  mental  processes. 
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Because  of  the  complexity  and  multi-dimensional  nature  of  the  dynamic  mental  model, 
any  single  construct  or  any  single  measure  of  a  construct  is  unlikely  to  capture  sufficient 
information  about  the  impact  of  a  new  technology.  Instead,  we  propose  that  the  CSE  evaluation 
should  include  at  least  one,  and  preferably  multiple  measures  of  each  of  situation  awareness, 
workload,  and  task-relevant  performance.  Use  of  multiple  measures  will  allow  for  a  richer  and 
more  accurate  answer  to  the  question  “how  does  the  new  technology  affect  the  human-machine 
interaction”?  Below,  we  provide  a  detailed  example  of  a  previous  M&S  evaluation  carried  out 
by  the  CACR  (in  conjunction  with  CMC  Electronics)  that  illustrates  the  importance  of  having 
multiple  constructs  and  measures.  Although  this  evaluation  was  conducted  prior  to  the 
development  of  the  CSE  framework,  it  has  characteristics  that  are  consistent  with  the  CSE 
approach  that  we  are  recommending. 

I.B  -  Example:  Direct  Voice  Input  for  the  CH146  Griffon  Helicopter 

In  designing  and  selecting  measures  (whether  of  performance,  SA,  or  workload),  it  is 
important  that  these  be  constrained  both  by  the  specific  situation  (e.g.,  the  CH146  environment) 
and  the  nature  of  the  technological  upgrade.  Thus,  the  first  step  in  an  M&S  evaluation  is  to  do  a 
rational  analysis  of  how  the  technology  is  expected  to  influence  pilot’s  behaviour.  In  the 
evaluation  of  direct  voice  input  (DVI)  for  the  Griffon,  heads-up  time  was  identified  as  a  key 
variable  that  should  be  affected  when  pilots’  were  given  DVI  capabilities.  Furthermore,  heads- 
up  time  is  an  ecologically  valid  measure,  because  pilots  are  instructed  and  trained  to  spend  as 
much  time  as  possible  looking  outside  of  the  aircraft.  The  DVI  interface  that  was  designed  for 
the  Griffon  allowed  the  non-flying  pilot  to  use  voice  commands  for  certain  common  actions  that 
normally  would  be  entered  on  the  onboard  computer  (the  CDU)  that  is  located  between  the  seats 
in  the  cockpit.  For  example,  pilots  could  change  the  radio  frequency  by  saying  “SET  RADIO  1 
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TO  121.5”  instead  of  typing  a  series  of  commands  on  the  CDU.  Using  a  voice  command  meant 
that  the  non-flying  pilot  could  potentially  keep  his  or  her  eyes  on  the  outside  world.  Because 
pilots  are  instructed  to  look  outside  the  cockpit  as  much  as  possible,  any  increase  in  heads-up 
time  would  be  a  reasonable  and  logical  outcome  of  the  change  from  manual  input  to  DVI. 
Heads-up  time  was  measured  in  this  evaluation  by  tracking  the  pilots’  head  position  throughout 
the  simulated  recce  missions,  and  then  comparing  the  total  heads-up  time  in  the  DVI  condition  to 
that  in  the  manual  input  position.  As  predicted,  heads-up  time  increased  with  DVI  relative  to  the 
manual  condition  (by  an  impressive  42%),  indicating  that  the  technology  change  had  at  least  one 
of  the  expected  and  desirable  outcomes  on  pilots’  behaviour. 

Heads-up  time  is  not  the  whole  story,  however.  Although  the  CSE  framework  shown  in 
Figure  1  had  not  been  developed  when  the  DVI  study  was  planned,  the  researchers  who 
conducted  the  evaluation  recognized  that  multiple  measures  of  behaviour  would  provide  a  more 
complete  picture  of  the  impact  of  DVI  on  the  human-machine  interaction.  In  particular, 
introduction  of  DVI  to  the  Griffon  cockpit  had  a  variety  of  other  potential  effects.  First,  DVI  had 
the  potential  to  change  the  CRM  in  that  the  flying  pilot  now  had  the  opportunity  to  interact  with 
the  CDU,  whereas  in  the  manual  input  situation  only  the  non-flying  pilot  can  enter  commands  on 
the  CDU.  So  the  workload  or  performance  of  the  flying  pilot  might  also  be  affected  by  the  new 
technology.  Second,  if  having  to  look  at  the  CDU  to  enter  commands  was  a  workload-intensive 
activity  for  the  nonflying  pilot,  then  DVI  might  decrease  his  or  her  overall  workload.  Other 
aspects  of  the  pilots’  task  performance,  however,  were  unlikely  to  be  affected  in  the  types  of 
missions  that  were  flown. 

Workload  was  measured  in  this  experiment  using  detection  of  auditory  and  visual  stimuli 
(i.e.,  targets)  by  both  pilots.  The  targets  (auditory  tones  or  visual  flashes  in  the  external  scene) 
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were  presented  randomly  in  the  course  of  the  simulated  missions.  Pilots  were  instructed  to 
respond  as  quickly  as  possible  when  they  detected  a  target  by  pressing  a  key.  Target  detection  as 
an  index  of  workload  has  been  used  extensively  and  thus  has  both  empirical  and  theoretical 
support.  Essentially,  the  speed  with  which  the  pilot  responds  to  the  target  can  be  used  as  a 
measure  of  his  or  her  available  mental  capacity.  In  the  DVI  experiment,  the  workload  of  the 
non-flying  pilot  was  less  in  the  DVI  condition  that  in  the  manual  condition,  suggesting  that  the 
DVI  facility  was  likely  to  result  in  improved  pilot  performance.  Importantly,  however,  the 
workload  of  the  flying  pilot  increased  significantly  in  the  DVI  condition.  Even  though  the  flying 
pilots  infrequently  used  the  DVI  capability,  the  incremental  change  to  their  responsibilities  was 
evident  in  the  workload  measure.  In  accord  with  the  CSE  framework,  DVI  had  a  significant 
impact  on  the  flying  pilot’s  dynamic  mental  model.  This  unexpected  finding  speaks  to  the 
complexity  of  changing  the  CRM  as  a  function  of  the  technological  upgrades  and  to  the 
importance  of  measuring  multiple  aspects  of  the  overall  pilot  activities. 

Situation  awareness  was  not  directly  measured  in  the  DVI  experiment.  Instead,  it  was 
inferred  that  SA  would  be  better  as  heads-up  time  increased  and  workload  decreased  (Vidulich, 
2000).  Because  inclusion  of  DVI  capability  resulted  in  a  substantial  increase  in  heads-up  time 
and  a  decrease  in  workload  for  the  nonflying  pilot,  it  was  concluded  that  the  SA  of  nonflying 
pilots  was  likely  to  improve  given  DVI  capabilities.  In  contrast,  the  SA  of  the  flying  pilot  may 
be  worse  in  DVI  conditions,  at  least  when  they  are  allowed  to  use  the  DVI  for  commands  that 
normally  would  be  handled  by  the  nonflying  pilot.  In  summary,  even  though  the  DVI  was 
conducted  before  our  fonnalization  of  the  CSE  framework,  it  provides  a  concrete  example  of 
how  multiple  measures  and  converging  evidence  is  necessary  in  the  CSE  process.  Actual 
assessment  of  situation  awareness  would  have  provided  an  even  more  comprehensive  picture  of 
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how  DVI  influenced  cockpit  activities. 

I.C  -  Implications  of  the  CSE  Framework  for  Cockpit  Research 

Our  review  of  the  literature  on  cockpit  research  shows  that  workload,  SA,  and  task-relevant 
performance  capture  a  substantial  amount  of  variability  in  the  human-machine  interaction. 
Workload  refers  to  the  cognitive  effort  required  by  the  pilot,  SA  is  the  pilot’s  perception  of 
events  and  integration  of  those  events  into  a  coherent  understanding  of  the  situation,  and  task¬ 
relevant  perfonnance  captures  the  behaviours  associated  with  the  interactions  between  the 
human  and  the  machine.  Direct  evaluation  of  performance  provides  important  infonnation 
about  the  pilot-system  interface  at  the  level  of  perceptual-motor  coordination  and  thus  allows  for 
the  evaluation  of  direct  interaction  with  the  system  through  systems  control.  Both  SA  and 
workload  represent  an  aspect  of  the  pilot-system  interaction  that  does  not  directly  reflect 
performance.  Therefore,  under  some  conditions,  we  should  not  expect  a  high  correlation  between 
SA  or  workload  and  the  overall  pilot-system  interaction.  Good  SA  does  not  always  lead  to  good 
performance  and  high  workload  does  not  always  predict  poor  performance.  Nevertheless,  these 
constructs  are  related  and  in  many  situations,  good  SA  will  predict  good  performance.  More 
generally,  the  CSE  framework  emphasizes  the  importance  of  multiple  interacting  aspects  of  pilot 
behaviour. 

Furthermore,  the  CSE  framework  makes  an  important  distinction  between  the  concepts  that 
are  being  measured  as  part  of  the  pilots’  behaviour  and  concepts  that  are  being  measured  as  the 
underlying  cognitive  mechanism.  One  of  the  major  problems  in  properly  defining  SA  is  a  lack  of 
distinction  between  SA  as  the  measured  construct  and  SA  as  the  pilot’s  internal  representation  of 
the  world  (e.g.,  Endsley,  2000).  SA  originated  as  a  description  of  behaviour  (i.e.,  that  some 
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pilots  are  much  more  skilled  than  other  pilots)  but  has  often  been  used  in  the  literature  as  the 
explanation  for  good  performance  (e.g.,  because  the  pilot  has  good  SA,  she  perfonned  well). 
This  confusion  between  SA  as  an  explanation  and  SA  as  a  description  needs  to  be  avoided  so 
that  SA  does  not  become  circular  and  therefore  an  empty  concept  (Flach,  1995).  . 

A  central  tenet  of  the  CSE  framework,  therefore,  is  that  the  human-machine  interaction  in 
the  cockpit  is  too  complex  for  a  single  concept  to  provide  sufficient  information  to  evaluate  the 
impact  of  an  interface  on  pilots’  overall  behaviour.  By  distinguishing  among  SA,  workload,  and 
performance  and  the  underlying  mental  representation  (the  dynamic  mental  model),  researchers 
can  more  clearly  operationalize  the  concepts  for  empirical  purposes.  Research  that  is  conducted 
within  the  CSE  framework  should  allow  for  comprehensive  and  valid  assessment  of  human 
factors  aspects  of  new  aviation  technology. 
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Section  II:  Definitions  and  Literature  Review 

Reviews  of  situation  awareness,  workload,  and  related  issues  in  cockpit  research  were 
done  with  various  databases,  including  PsycINFO,  Social  Science  Citation  Index  (SSCI), 
CISTISource  and  Web  of  Science.  Pertinent  references  from  other  sources  (e.g.,  conference 
proceedings  of  the  Aviation  Psychology  conference)  were  also  included.  The  edited 
bibliography  of  articles  gathered  via  these  searches  is  included  at  the  end  of  this  initial  report.  A 
reference  list  that  includes  those  cited  in  the  body  of  this  report  is  also  included.  Our  goal  with 
the  review  portion  of  this  report  is  to  provide  a  critical  overview  of  these  psychological 
constructs,  and  to  give  recommendations  that  are  pertinent  to  modeling  and  simulation  activities 
relevant  to  the  CHI 46.  A  third  section  (to  be  added  in  the  final  version  of  the  report)  will 
include  descriptions  of  the  results  of  Experiments  1  through  3,  where  the  CSE  framework  was 
used  to  develop  and  implement  the  human  factors  component  of  the  M&S  process. 

II. A.  -  Situation  Awareness 

II.A.l.  -  Defining  Situation  Awareness 

Situation  awareness,  defined  simply  as  “knowing  what  is  going  on  around  you”  (Endsley, 
2000,  p.  5)  is  a  construct  that  was  proposed  originally  to  capture  why  some  fighter  pilots  were 
more  successful  (and  therefore  lived  longer)  than  others  (Spick,  1988).  As  discussed  by  Endsley 
(2000),  SA  is  closely  tied  to  knowing  how  to  distinguish  important  information  in  the 
environment  from  less  important  information  and  to  the  ability  to  quickly  comprehend  the 
importance  of  changes  to  elements  in  the  environment  (Adams,  Tenney,  &  Pew,  1995;  Durso  & 
Gronlund,  1999;  Endsley,  1995;  2000;  Sarter  &  Woods,  1995).  Spick  (1988)  suggests  that  an 
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important  aspect  of  what  made  the  aces  better  than  the  average  pilot  was  their  superior  judgment 
about  when  to  either  engage  in  a  fight  or  disengage  in  one  that  was  not  going  well.  Hence,  SA 
represents  pilots’  ability  to  keep  track  of  objects  such  as  enemy  aircraft  in  the  environment  and  to 
estimate  the  probability  of  success  in  a  given  situation. 

Situation  awareness  is  appealing  because  it  seems  intuitively  obvious  that  successful 
completion  of  missions  requires  that  pilots  are  aware  of  the  status  of  the  aircraft  and  the  various 
elements  of  the  situation.  It  has  proven  difficult,  however,  to  define  what  SA  is  for  empirical 
purposes.  Researchers  offer  different  perspectives  on  what  constitutes  SA.  This  lack  of  a  clear 
definition  has  made  it  difficult  to  establish  a  coherent  framework  for  conducting  SA  research. 
Furthermore,  researchers  disagree  as  to  whether  or  not  SA  is  not  a  unitary  phenomenon  (Pew, 
2000;  Sarter  &  Woods,  1991).  Researchers  face  a  variety  of  difficulties  in  defining  SA.  First, 
SA  encompasses  a  wide  variety  of  factors  such  as  keeping  track  of  objects  in  the  environment, 
attending  to  relevant  infonnation,  and  pilot’s  knowledge  of  the  mission.  Second,  the  elements  of 
the  situation  that  are  relevant  vary  considerably  depending  on  the  nature  of  the  mission  and  the 
aircraft.  For  example,  although  altitude  tracking  is  critical  to  a  helicopter  pilot  attempting  to 
maintain  a  hover  over  the  ocean,  altitude  tracking  is  not  critical  to  a  fighter  pilot  who  is  engaged 
with  an  enemy  aircraft.  These  complexities  have  made  it  difficult  to  produce  a  single,  definitive 
description  of  SA. 

One  commonality  across  different  descriptions  of  SA  is  an  emphasis  on  the  underlying 
cognitive  mechanisms  that  determine  SA  and  that  are  common  to  different  situations  (Adams  et 
ah,  1995;  Endsley,  2000).  Endsley  (1988)  defined  SA  as:  “...the  perception  of  the  elements  in 
the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the 
projection  of  their  status  in  the  future”  (1988,  p.  97).  Endsley  included  three  levels  within  the 
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construct  of  SA:  perception,  comprehension  and  projection  (Endsley,  1995,2000).  Endsley’s 
model  will  be  considered  in  some  detail  because  it  is  the  most  widely  cited  description  of  the 
construct  of  SA. 

Level  1  SA  is  the  operator’s  perception  of  the  current  status  of  the  environment.  For 
instance,  in  a  cockpit  environment,  Level  1  SA  includes  the  perception  of  the  various  indicators 
on  the  display  such  as  the  altitude  indicator,  heading,  and  relative  position.  Level  1  also  includes 
the  perception  of  objects  in  the  environment  such  as  mountains,  trees,  other  aircraft,  and  warning 
lights.  Level  2  SA  refers  to  how  the  situation  is  understood  (i.e.,  how  the  various  elements  that 
are  perceived  at  the  earlier  level  are  integrated  and  comprehended).  At  Level  2  SA  the  various 
elements  perceived  in  the  situation  are  integrated  in  light  of  the  pilot’s  goal  and  the  requirements 
of  the  task.  At  this  level  the  pilot  forms  a  holistic  picture  of  the  environment  that  combines  the 
individual  elements  with  stored  knowledge  of  these  elements.  Level  3  SA  refers  to  the  pilot’s 
ability  to  predict  future  status  of  the  situational  elements  and  to  anticipate  the  requirements  of  the 
operating  system.  This  is  the  highest  level  of  SA  and  is  based  on  the  elements  of  both  Levels  1 
and  2. 

Endsley  (1995)  proposed  a  theoretical  model  of  SA.  In  her  model  Endsley  describes  SA  as  a 
dynamic  mental  model  of  the  world  that  is  detennined  by  various  cognitive  factors  such  as 
attention,  working  memory,  and  prior  knowledge.  Unfortunately  Endsley’s  model  is  far  too 
complex  to  generate  testable  hypotheses  or  guide  experiments.  Also,  it  is  not  clear  within  the 
model  whether  measured  SA  is  the  behaviour  of  the  pilot  or  the  underlying  cognitive  state 
(Flach,  1995).  Furthermore,  Endsley  emphasizes  that  SA  as  a  cognitive  state  that  is  independent 
from  perfonnance.  Therefore,  performance  measures  often  play  a  small  role  in  cockpit 
evaluations  unless  specifically  defined  as  being  an  indirect  measure  of  SA.  In  summary, 
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although  Endsley’s  model  has  had  considerable  impact  on  the  field  of  aviation  psychology,  it 
functions  more  as  a  way  of  describing  the  human-machine  interaction,  rather  than  as  a 
framework  for  developing  M&S  research. 


II.A.2  -  Measuring  Situation  Awareness 

In  combination,  Pew  (2000)  and  Vidilich  (2000)  proposed  five  broad  categories  of 
measurement  that  have  been  used  in  research  on  situation  awareness.  As  shown  in  Table  2.1, 
these  include  verbal  protocols,  awareness  queries,  subjective  assessments,  performance 
assessments,  and  situation  manipulations.  Pew  (2000)  provides  the  more  comprehensive 
taxonomy  of  SA  measures,  however,  Vidulich  (2000)  noted  that  researchers  have  often  inferred 
SA  from  performance,  rather  than  measuring  it  directly.  In  accord  with  the  proposed  CSE 
framework,  measuring  only  SA  or  only  workload  is  likely  to  be  restrictive.  The  overlap  between 
SA  and  performance  noted  by  Vidulich  supports  our  view  that  multiple  constructs  and  multiple 
measures  of  these  constructs  are  most  likely  to  provide  a  comprehensive  overview  of  the  human- 
machine  interaction. 
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Table  2.1 


Taxonomy  of  Measurement  for  Situation  Awareness  (after  Pew,  2000,  and  Vidulich,  2000) 


Our 

terminology 

Pew’s 

Terminology 

[Vidulich]3 

Description 

Verbal  Protocols 

Verbal  protocols 

Pilots  provide  online  or  immediately  retrospective  think- 

aloud  descriptions  of  their  mental  processes 

Situation 

Manipulations 

Direct  system 

perfonnance 

measures 

During  the  simulated  mission,  the  experimenter: 

(a)  introduces  either  an  anomaly,  or  some  kind  of  subtle 

scenario  manipulation  that  the  pilot  is  expected  to  detect  if 

they  have  good  SA 

(b)  introduces  disruptions  that  pilots  must  recover  from 

(e.g.,  freeze  the  simulation,  then  introduce  an  offset  to  the 

heading) 

(c)  introduces  anomalous  data  or  instrument  readings  and 

observes  pilots’  responses 

Awareness 

Queries* 

Direct 

experimental 

techniques 

[memory  probe 

measures] 

Pilots  respond  to  queries  or  probes  collected  when  a 

simulation  is  frozen  (e.g.,  best  known  method  is  SAGAT; 

Endsley  [2000]) 

Subjective 

Assessment* 

Subjective 

measures 

Self-assessments  (usually  after  the  mission  has  finished), 

expert  judgments  of  pilot  SA,  peer  ratings,  supervisor  or 

instructor  ratings 

Performance 

Assessment* 

Experimenter  makes  an  inference  about  effects  of  a 

technology  on  SA  by  evaluating  changes  in  performance 

a  terminology  usee 

l  by  Vidulich  (2000)  if  it  is  different  than  that  used  by  Pew  (2000) 

*  categories  included  in  Vidulich  (2000)  meta-analysis. 
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II.A.3  -  Summary 

SA  represents  how  well  the  pilot  perceives  the  situational  elements  and  integrates  them  into 
a  coherent  understanding  of  the  situation.  Measurement  of  SA  can  show  how  a  well-designed 
system  supports  the  pilots’  cognitive  needs  whereas  a  poorly  designed  system  may  result  in 
pilots  lacking  a  coherent  understanding  of  their  surroundings  (Jones  &  Endsley,  1996).  However, 
although  SA  provides  important  insight  into  some  aspects  of  the  pilot-system  interaction,  other 
aspects  may  be  neglected.  For  example,  by  focusing  on  measuring  a  pilot's  understanding  of  the 
situation,  some  critical  aspects  of  behaviour  may  be  overlooked.  For  example,  perceptual-motor 
coordination  that  is  reflected  in  how  well  the  pilot  maintains  a  route  or  reacts  to  stimuli  in  the 
environment  is  an  important  part  of  the  overall  adaptation  to  the  requirements  of  the  cockpit 
system.  Thus,  measurement  of  SA  needs  to  be  supplemented  by  other  sources  of  information. 

The  proposed  CSE  framework  would  help  researchres  to  capture  more  of  the  variance  involved 
in  pilot-system  interaction  as  compared  to  measuring  a  single  construct.  Thus,  the  goal  of  the 
CSE  framework  is  to  guide  future  research  in  this  domain  based  on  the  assumption  that  multiple 
measures  of  three  central  constructs  (i.e.,  SA,  workload,  and  task-relevant  performance),  will 
give  a  reasonably  complete  perspective  on  how  any  given  technology  influences  human 
responses  in  the  cockpit. 

II.B.  -  Workload 

II.B.l  -  Defining  Workload 

The  concept  of  mental  workload  has  been  used  in  the  aviation  community  to  capture  the 
relations  among  task  difficulty,  limitations  in  pilots’  cognitive  abilities,  and  performance  (Flach 
&  Kuperman,  2001;  Wickens,  2001).  Workload  is  an  index  of  how  much  cognitive  effort  is 
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required  by  pilots  as  they  interact  with  a  given  system.  Measuring  workload  is  particularly 
relevant  to  cockpit  research  where  the  challenge  is  to  reach  a  balance  between  the  quantity  of 
information  provided  and  the  pilots’  capacity  to  process  that  information.  In  general,  as  more 
effort  is  required  from  the  pilot  for  a  given  task,  fewer  cognitive  resources  are  available  for 
accomplishing  other  tasks.  High  workload  is  associated  with  emotional  stress,  fatigue,  and 
decrements  in  performance  (Hart  &  Wickens,  1990;  Tsang  &  Wilson,  1997). 

Workload  has  been  defined  as:  “a  hypothetical  construct  that  represents  the  cost  incurred  by 
a  human  operator  to  achieve  a  particular  level  of  performance”  (Hart  &  Staveland,  1988,  p.  140). 
This  definition  of  workload  is  framed  within  the  resource  model  of  information  processing  such 
that  workload  represents  the  relation  between  the  availability  of  cognitive  resources  and  the 
demands  of  the  task  (Wickens,  1991).  Within  this  model,  performance  decrements  occur  when 
tasks  that  are  being  performed  simultaneously  require  the  same  cognitive  resources.  This 
resource-based  definition  of  workload  is  consistent  with  a  large  literature  on  the  relations 
between  mental  effort  and  performance  in  a  wide  range  of  cognitive  tasks.  In  aviation  research, 
however,  a  disadvantage  of  this  definition  of  workload  is  that  it  does  not  distinguish  between 
how  the  pilot  experiences  the  level  of  task  difficulty  and  the  actual  impact  it  has  on  perfonnance. 

In  the  domain  of  aviation,  researchers  have  placed  more  emphasis  on  the  pilot's  experience 
of  workload  (i.e.,  subjective  workload)  than  on  objective  workload  (i.e.,  performance  trade-offs) 
by  measuring  physiological  responses  and  asking  pilots  to  rate  their  perceived  workload  (Hart  & 
Wickens,  1990;  Wickens,  1999).  The  focus  on  how  pilots  experience  workload  reflects  the 
finding  that,  in  highly  trained  pilots,  high  workload  does  not  necessarily  reflect  poor 
performance  (Vidulich  &  Wickens,  1986;  Yeh  &  Wickens,  1988).  Nevertheless,  if  the  pilot 
experiences  a  task  as  demanding  (i.e.,  high  in  workload),  he  or  she  may  become  fatigued  and 
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mission  effectiveness  may  be  compromised.  A  common  pattern  seen  in  workload  research  is  that 
performance  is  stable  for  a  long  time  and  then  suddenly  declines.  Initially,  researchers  assumed 
that  the  performance  decrement  reflected  a  sudden  increase  in  workload  that  exceeded  the  pilots’ 
cognitive  resources.  However,  subjective  ratings  and  physiological  measures  showed  that  pilots 
actually  experienced  a  steady  increase  in  workload  up  to  the  point  of  the  perfonnance  decrement 
(Andre,  2001). 

In  summary,  both  the  objective  workload  (i.e.,  trade-offs  between  performance  and  effort) 
and  subjective  workload  (i.e.,  the  pilots’  perception  of  workload)  are  important  aspects  of  the 
human-machine  interface  in  research  on  aviation.  However,  although  research  suggests  that 
workload  measures  are  a  valuable  tool  in  aviation  research,  they  have  not  been  used  consistently 
because  there  is  no  consensus  in  how  workload  should  be  measured  (Flach  &  Kuperman,  2001; 
Wickens,  2001). 

II. B. 2  -  Measuring  workload 

Numerous  measures  have  been  proposed  to  evaluate  workload  in  the  cockpit.  The 
proposed  measures  of  workload  can  be  classified  into  four  broadly  defined  categories;  primary- 
task  measures,  secondary-task  measures,  physiological  measures,  and  subjective  ratings 
(Bortolussi,  Kantowitz  &  Hart,  1986;  Casali  &  Wierwille,  1984;  Hart  &  Wickens,  1990; 

Wickens  &  Hollands,  1999). 

Primary-task  measures.  Measuring  performance  on  the  primary  task  (i.e.,  the  task  required 
by  the  system  in  question)  allows  for  the  assessment  of  whether  the  task  causes  boredom  and 
hence  less  vigilance  over  time,  whether  perfonnance  is  stable  over  time,  and  at  what  point 
performance  breaks  down.  However,  performance  on  a  primary  task  is  rarely  used  to  evaluate 
workload  as  it  tends  not  to  co-vary  with  pilots’  experience  of  workload.  Therefore,  poor 
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performance  may  or  may  not  reflect  demands  on  resources  (Yeh  &  Wickens,  1988).  Thus, 
primary  task  perfonnance  can  be  considered  a  baseline  measure,  but  it  is  only  directly  indicative 
of  workload  when  cognitive  resources  are  exceeded  and  perfonnance  begins  to  break  down. 

Secondary-  task  measures.  When  pilots  simultaneously  perform  two  tasks,  the  primary  task 
is  the  central  aviation  task  (e.g.,  hovering)  whereas  the  secondary  task  is  added  by  the  researcher 
to  reflect  the  availability  of  cognitive  resources.  For  example,  in  the  DVI  evaluation  reported  by 
Herdman  et  al.  (2001),  the  primary  task  was  to  complete  the  mission  whereas  the  secondary  task 
was  to  detect  the  auditory  and  visual  targets.  The  pilot  is  instructed  to  perfonn  as  well  as 
possible  on  the  primary  task  and  allocate  any  leftover  resources  to  the  secondary  task.  As  the 
primary  task  becomes  more  difficult,  fewer  resources  are  available  for  performing  the  secondary 
task  and  thus  the  focus  is  on  decrements  in  perfonnance  on  the  secondary  task.  Common 
secondary  tasks  used  to  evaluate  workload  are:  a)  a  rhythmic  tapping  task  where  the  pilot  must 
produce  a  finger  or  a  foot  tap  at  a  constant  rate,  b)  random  number  generation  where  the  pilot 
must  randomly  generate  numbers,  and  c)  reaction  time  to  probe  stimuli  (e.g.,  Herdman  et  al., 
2001). 

Secondary  task  measures  have  been  used  frequently  to  evaluate  workload.  However,  the 
method  has  some  limitations.  First,  when  the  primary  task  reaches  a  certain  level  of  difficulty  the 
pilots  may  simply  abandon  the  secondary  task.  Second,  research  on  workload  using  secondary 
measures  has  shown  that  different  types  of  secondary  measures  will  be  interfered  with 
selectively  by  the  primary  measure.  For  example,  tapping  is  more  likely  to  interfere  with  a 
spatial  primary  task  than  with  a  verbal  primary  task  (Baddeley  &  Logie,  1999).  This  selective 
interference  means  that  the  workload  differences  caused  by  the  primary  task  may  be 
underestimated  if  the  primary  and  secondary  tasks  require  different  processing  resources  (see 
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Hart  &  Wickens,  1990).  Third,  introducing  a  secondary  task  to  the  pilot  may  be  intrusive  for  the 
pilot.  Despite  these  limitations,  however,  secondary  task  measures  provide  a  very  useful 
objective  index  of  pilot  workload.  Furthennore,  the  use  of  the  dual-task  method  is  a  well- 
established  way  of  indexing  cognitive  demands  in  the  wider  literature  (Baddeley  &  Logie,  1999) 
and  thus  considerable  research  can  be  accessed  to  develop  and  interpret  the  results  of  workload 
research  in  the  aviation  field. 

Physiological  measures.  Physiological  measures  of  workload  include  heart  rate,  eye  blink 
rate,  pupil  diameter,  respiration  frequency,  blood  pressure,  and  electrical  activity  of  the  brain. 
Use  of  these  measures  is  based  on  the  assumption  that,  for  example,  an  increase  in  heart  rate  or 
respiration  reflects  a  concomitant  (but  not  necessarily  conscious)  increase  in  workload.  One 
advantage  of  physiological  measures  is  that  they  are  less  obtrusive  than  subjective  ratings  or 
secondary  task  measures.  Furthermore,  measurement  of  physiological  responses  provides 
information  about  the  pilots’  emotional  and  physical  activation  during  the  course  of  a  task  as 
well  as  their  processing  time  and  cognitive  load.  Researchers  have  shown  that  physiological 
measures  are  a  reliable  indication  of  workload  (Bortolussi,  Kantowitz  &  Hart,  1986;  Casali  & 
Wierwille,  1984).  However,  the  main  limitation  of  physiological  measures  is  that  they  are 
indirect  indices  of  how  the  pilot  actually  experiences  the  workload.  Furthennore,  physiological 
measures  may  not  relate  directly  to  perfonnance. 

Subjective  evaluation.  Subjective  ratings  of  workload  have  been  used  most  frequently  in 
research  on  aviation.  They  have  the  advantage  of  directly  measuring  the  pilots’  experience  of 
workload  in  tenns  of  cognitive  cost  and  attentional  resources  (Hart  &  Wickens,  1990).  Two 

2  Although  visual  scanning  or  other  measures  of  behavior  are  sometimes  classified  as  “physiological  measures”,  in 
the  CSE  framework  these  would  be  considered  as  elements  of  “task-relevant  performance”.  Visual  scanning  is  a 
very  indirect  measure  of  physiological  processes. 
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questionnaire  measures  are  commonly  used  in  evaluating  the  pilots'  experience  of  workload,  the 
NASA  task  load  index  (NASA  TLX;  Hart  &  Staveland,  1988)  and  the  subjective  workload 
assessment  technique  (SWAT;  Reid  &  Nygren,  1988).  The  NASA  TLX  assesses  workload  on 
five  7-point  scales;  mental  demand,  physical  demand,  temporal  demand,  perfonnance,  effort,  and 
frustration  level.  The  SWAT  assesses  workload  on  three  3-point  scales;  time  load,  mental  effort, 
and  stress. 

Subjective  ratings  of  workload  are  widely  used.  Subjective  ratings  are  relatively  easy  to 
administer  and  have  minimal  impact  on  pilots’  ongoing  behaviour  because  they  are  typically 
given  after  a  task  is  completed.  These  measures  are  limited,  however,  because  they  are  rarely 
used  to  index  online  workload  (i.e.,  workload  that  is  experienced  while  the  pilot  is  doing  the 
task)  and  because  they  are  subjective.  As  with  any  measure  that  requires  pilots  to  introspect  on 
their  behaviour,  there  are  always  questions  about  whether  such  measures  are  reliable  (i.e.,  are  the 
measures  consistent?  Can  pilots’  calibrate  their  responses  across  tasks?)  and  valid  (i.e.,  do  the 
responses  actually  reflect  workload?;  Wickens,  1999).  Furthermore,  rating  scales  are  limited  in 
that  they  will  only  tap  into  a  subset  of  factors  that  may  be  relevant  to  the  overall  experience  of 
workload  (Hart  &  Wickens,  1990).  More  research  in  which  subjective  measures  are  collected  in 
conjunction  with  objective  measures  is  needed  to  probe  the  validity  of  these  assessments. 

II.B.3  -  Summary 

In  aviation  research,  assessment  of  workload  is  important  for  evaluating  the  adequacy  and 
feasibility  of  the  human-machine  interaction.  High  workload  may  reflect  a  poorly  designed 
system  that  puts  an  unnecessary  load  on  the  pilot.  For  example,  an  instrument  layout  in  the 
cockpit  that  requires  longer  gaze  time  by  pilots  than  an  alternative  layout  may  increase  pilot 
workload.  Similarly,  Herdman  et  al.  (2001)  found  that  the  addition  of  a  DVI  system  to  the 
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CHI 46  cockpit  resulted  in  an  increase  in  the  workload  of  the  flying  pilot  (as  indexed  by  the 
secondary  task  performance).  Hence,  it  is  crucial  to  measure  pilot  workload  in  any  technology 
evaluation.  Although  four  categories  of  workload  measurement  have  been  described  (i.e., 
primary-task  measures,  secondary-task  measures,  physiological  measures  and  subjective  ratings), 
multiple  measures  have  rarely  been  used  in  the  same  studies.  Furthermore,  workload  measures 
have  not  been  used  consistently  in  cockpit  research  (Wickens,  2001).  As  with  performance  and 
situation  awareness,  converging  measures  of  workload  are  most  likely  to  provide  a 
comprehensive  understanding  of  the  impact  of  technology  on  the  cockpit  activity. 
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ANNEX  B 

Pilot  Questionnaires 
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TAMSS  Experiment  1  Participant  ID: 

Background  Information 

1 .  Age: _  (in  years) 

2.  Years  in  CF: _ 

3.  Years  as  a  pilot: _ 

4.  Handedness:  _ left  OR _ right 


Please  Estimate  your  flight  thime  in  each  category: 


Type  of  Flight 

Estimated  time  (in  hours) 

1.  Total  flight  time 

2.  Total  rotary  wing 

3.  Total  Griffon 

4.  Total  NVG 

5.  Total  NVG  with  HUD 

6.  Time  spent  using  a  flight  simulator 

Specify  the  type  of  simulator  experiences  that  you  have  had: 
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TAMSS  Experiment  1  Participant  ID:  _ 

HUD  vs.  no-HUD  comparison 

Rate  the  impact  of  having  a  HUD  as  compared  to  NO  HUD  on  your  experiences  in  the  simulator. 
Circle  your  response  for  each  item.  A  rating  of  ‘4’  means  there  was  no  difference  between  the 
HUD  and  no-HUD  conditions. 


COMPARISON 

HUD  much 
better  than 
no-HUD 

No  difference 

No-HUD 

much  better 
than  HUD 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

2.  Awareness  of  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

3.  Awareness  of  altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

4.  Awareness  of  airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

5.  Awareness  of  spatial 

1 

2 

3 

4 

5 

6 

7 

NA 

orientation 

6.  Awareness  of  activity  on  the 

1 

2 

3 

4 

5 

6 

7 

NA 

ground 

7.  Awareness  of  activity  in  the 

1 

2 

3 

4 

5 

6 

7 

NA 

air 

8.  Awareness  of  aircraft  systems 

1 

2 

3 

4 

5 

6 

7 

NA 

9.  Cross  checking  relevant 

1 

2 

3 

4 

5 

6 

7 

NA 

instruments/symbology 

10.  Using  information  from  the 

1 

2 

3 

4 

5 

6 

7 

NA 

scene  to  control  the  aircraft 

1 1 .  Eyes-out  time 

1 

2 

3 

4 

5 

6 

7 

NA 

12.  Low-level  flight 

1 

2 

3 

4 

5 

6 

7 

NA 

13.  Low-level  maneuvering 

1 

2 

3 

4 

5 

6 

7 

NA 

14.  Maintaining  airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

15.  Maintaining  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

16.  Maintaining  altitude 

1 

2 

3 

4 

5 

6 

7 

NA 
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TAMSS  Experiment  1 
Mission  Assessment 


Participant  ID 

Date 

Time 

Mission  number 

Scenario  Code 

Condition 

HUD  no-HUD 

Instructions 

If  you  have  additional  comments  about  any  question  or  item,  please  write  these  on 
the  back  of  the  corresponding  page. 
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Ml.  Rate  how  frequently  you  looked  down  at  the  instrument  panel  during  this  mission. 


never 

rarely 

sometimes 

often 

very  often 

M2.  Rate  how  well  you  performed  each  of  the  following  tasks  during  this  mission.  Circle 
N/A  if  the  item  was  not  applicable  to  this  mission. 


PERFORMANCE 

Task 

very  very 

poor  adequate  good 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  correct 

heading 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  correct 

altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  correct 

airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Cross  checking  relevant 

instruments/symbology 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  information  from 

the  external  scene  to 

control  the  aircraft 

1 

2 

3 

4 

5 

6 

7 

NA 
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M3.  Rate  the  difficulty  of  each  of  the  following  tasks  during  the  mission.  Circle  N/A  if 
the  item  was  not  applicable  to  this  mission. 


DIFFICULTY 

Task 

very  very 

easy  moderate  difficult 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  correct 

heading 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  correct 

altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  correct 

airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Cross  checking  relevant 

instruments/symbology 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  information  from 

the  external  scene  to 

control  the  aircraft 

1 

2 

3 

4 

5 

6 

7 

NA 

M4.  Rate  the  difficulty  of  getting  (reading)  information  from  each  of  the  following  sources 
during  this  mission.  Circle  NA  if  not  applicable  to  this  mission. 


DIFFICULTY 

very  very 

easy  moderate  difficult 

Heading 

1 

2 

3 

4 

5 

6 

7 

NA 

RAD  Alt 

1 

2 

3 

4 

5 

6 

7 

NA 

Airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Attitude 

1 

2 

3 

4 

5 

6 

7 

NA 

External  Scene 

1 

2 

3 

4 

5 

6 

7 

NA 
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M5.  Rate  your  awareness  of  the  following  during  the  mission.  Circle  NA  if  not  applicable 
to  this  mission. 


AWARENESS 

very  moderate  very 

low  high 

Overall  Situation 

Awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

Heading 

1 

2 

3 

4 

5 

6 

7 

NA 

RAD  Alt 

1 

2 

3 

4 

5 

6 

7 

NA 

Activity  on  the  ground 

1 

2 

3 

4 

5 

6 

7 

NA 

Activity  in  the  air 

1 

2 

3 

4 

5 

6 

7 

NA 

Airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Attitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Spatial  Orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

Environmental  Events 

1 

2 

3 

4 

5 

6 

7 

NA 

Aircraft  Systems 

1 

2 

3 

4 

5 

6 

7 

NA 
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M6.  Rate  your  workload  during  this  mission.  Circle  NA  if  not  applicable  during  this  mission. 


WORKLOAD 

very  moderate  very 

low  high 

Between  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

At  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

When  reporting 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Overall 

1 

2 

3 

4 

5 

6 

7 

NA 

At  any  point,  did  you  find  that  your  workload  was  very  different  (e.g.,  much  higher  or  lower) 
than  across  the  mission  as  a  whole?  YES  or  NO 
If  yes,  please  elaborate: 
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ANNEX  C 

Sample  Mission  Routes 
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Executive  Summary 

As  part  of  the  Department  of  National  Defence  (DND)  Tactical  Aviation  Mission  System  Simulation 
(TAMSS)  Situation  Awareness  initiative,  the  Centre  for  Applied  Cognitive  Research  (CACR)  at  Carleton 
University  has  proposed  a  Cognitive  Systems  Engineering  (CSE)  framework  to  be  used  as  a  guide  for 
conducting  and  interpreting  evaluations  in  M&S  programs.  The  present  document  is  a  report  on 
Experiment  2  (E2)  of  the  TAMMS  SA  program.  Experiment  2  of  the  TAMSS  SA  program  had  two 
major  activities.  The  first  activity  was  to  integrate  the  DND  CH-146  ERSTA-like  system  with  the 
simulator  facility  at  the  Carleton  University  Centre  for  Applied  Cognitive  Research  (CACR).  The 
ERSTA  modelling  and  integration  activity  is  to  be  extended  into  the  third  and  final  TAMSS  SA 
experiment.  Thus,  the  report  on  this  activity  will  be  presented  as  part  of  the  Experiment  3  report.  The 
second  major  activity  in  Experiment  2  was  to  conduct  a  study  to  evaluate  the  impact  of  the  prototyped 
digital  moving  map  and  an  ERSTA-like  sensor  capability  on  CH-146  Griffon  aircrew.  The  particular 
focus  of  the  experiment  was  on  how  ERSTA-like  system  affects  the  performance,  situation  awareness  and 
workload  of  the  CH-146  Mission  Commander  (MC).  This  experimental  activity  is  reported  in  the  present 
document. 
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Situation  Awareness,  Performance,  and  Workload  in  Simulated  Flight: 
The  Impact  of  an  ERSTA-Like  System  on  the  CH-146  Mission  Commander 


This  document  is  a  report  on  Experiment  2  (E2)  of  the  TAMMS  SA  program.  This  experiment  had  two 
major  activities.  The  first  activity  was  to  integrate  the  DND  CEI-146  ERSTA-like  model1  with  the 
simulator  facility  at  the  Carleton  University  Centre  for  Applied  Cognitive  Research  (CACR).  This 
activity  included  (a)  modifying  the  extant  DND  ERSTA  model  to  reflect  the  core  mapping  and  sensor 
capabilities  of  the  ERSTA  system  that  is  anticipated  for  the  CEI-146  Griffon,  (b)  designing  and 
implementing  moving  map  and  sensor  display  interfaces  for  the  cockpit,  (c)  making  the  ERSTA-like 
system  compliant  with  Eligh-Level  Architecture  (EILA)  specifications,  and  (d)  linking  the  EILA-compliant 
ERSTA-like  model  with  the  CACR  Griffon  simulator.  The  ERSTA  modelling  and  integration  activity  is 
to  be  extended  into  the  third  and  final  TAMSS  SA  experiment.  Thus,  the  report  on  this  activity  will  be 
presented  as  part  of  the  TAMSS  Final  report. 

The  second  major  activity  in  Experiment  2  was  to  conduct  a  study  whereby  the  Cognitive  Systems 
Engineering  (CSE)  framework  that  has  been  proposed  by  the  Carleton  University  CACR  was  used  to 
evaluate  the  impact  of  the  prototyped  ERSTA-like  digital  moving  map  and  sensor  capability  on  CEI-146 
Griffon  aircrew.  The  particular  focus  of  the  experiment  was  on  how  the  ERSTA-like  system  affects  the 
performance,  situation  awareness  and  workload  of  the  CH-146  Mission  Commander  (MC).  This 
experimental  activity  is  reported  in  the  present  document. 

1  Introduction 
1.1  Goals 

The  experimental  goals  were  to  extend  the  evaluation  of  the  Cognitive  System  Engineering  (CSE) 
framework  proposed  by  the  CACR  for  evaluating  the  impact  of  novel  technology  on  aircrew  in  the 
CHI 46  Griffon  helicopter.  This  involved  (a)  integration  of  an  emulation  of  the  DND  CH-146  ERSTA- 
like  system  into  the  CACR  simulator  using  HLA  protocol,  (b)  development  of  realistic  tactical  scenarios, 
(c)  further  development  of  the  data  collection  capabilities  of  the  C  ACR  simulation  facility  and  (d)  further 
development  of  the  questionnaire  battery  used  for  assessing  workload. 

In  this  experiment,  aircrew  consisting  of  a  Flying  Pilot  (FP)  and  a  Mission  Commander  (MC)  completed  a 
series  of  zone  recce  missions.  Of  primary  interest  was  how  digital  moving  map  and  ERSTA-like  sensor 
capabilities  affect  the  performance,  situation  awareness  and  workload  of  the  CH-146  MC  while 
completing  these  missions.  Accordingly,  the  following  three  conditions  were  included  in  the  experiment: 

•  Paper  Map  (P-Map). 

This  is  a  baseline  condition  that  reflects  the  current  situation  in  the  Griffon  where  aircrew 
(i.e.,  the  MC)  navigate  using  a  hand-held  paper  map  and  detect  and  identify  targets 
without  aid  of  a  sensor. 

•  Moving  Map  (M-Map) 


1  An  ERSTA  system  for  the  CH-146  fleet  had  not  been  acquired  within  the  timeframe  of  this  study.  Therefore,  the 
ERSTA  system  that  was  used  in  this  study  was  intended  only  as  an  approximate  representation.  Although  an 
attempt  was  made  to  provide  the  general  functionality  of  the  anticipated  ERSTA  system,  the  full  ERSTA  capabilities 
(especially  of  the  sensor)  were  not  represented.  For  this  reason,  the  term  “ERSTA-like  system”  is  used  in  this 
document. 
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In  this  condition,  the  MC  was  provided  with  a  digital  moving  map  positioned  on  the  MC 
lap.  A  paper  map  was  also  provided  for  use  at  the  discretion  of  the  MC.  As  in  the  paper 
map  condition,  the  aircrew  were  required  to  detect  and  identify  targets  without  aid  of  a 
sensor. 

•  Moving  Map  plus  Sensor  (M-Map/Sensor) 

In  this  condition,  both  the  digital  moving  map  (and  the  paper  map)  and  the  ERSTA-like 
sensor  capability  were  provided.  The  ERSTA-like  sensor  (camera)  image  was  displayed 
on  the  front  centre  console,  i.e.,  where  the  current  Griffon  FLIR  image  is  normally 
located.  In  this  condition,  aircrews  were  able  to  use  the  sensor  image  to  support  target 
detection  and  identification. 

The  TAMSS  SA  project  is  focused  on  situation  awareness.  We  have  argued,  however,  that  assessing  SA 
in  isolation  will  not  provide  sufficient  evidence  to  allow  for  good  decisions  in  the  modelling  and 
simulation  process.  Accordingly,  the  present  experiment  was  designed  to  sample  all  three  dimensions  of 
behaviour  outlined  in  the  CSE  framework:  task-relevant  performance,  workload,  and  situation  awareness. 
Furthermore,  the  main  thrust  of  the  CSE  framework  is  to  provide  converging  measures,  allowing 
researchers  to  give  an  overall  perspective  of  performance  that  does  not  rely  solely  on  a  single  construct  or 
single  method  of  measurement.  To  this  end,  both  subjective  and  objective  measures  of  task-relevant 
performance  and  situation  awareness  were  obtained.  For  workload,  only  subjective  measures  were 
obtained. 

1.2  Overview  of  measures 

Task-relevant  Performance  -  In  this  experiment,  pilots  flew  realistic  zone-recce  missions  (see  descriptions 
in  Annex  A).  For  each  mission  a  primary  objective  was  defined  and  details  were  given  concerning  the 
flight  path  leading  to  the  Release  Point  (RP).  Pre-mission  information  was  given  concerning  known 
position  and  movement  of  friendly  and  enemies  forces  in  the  area.  Aircrew  planned  their  post  RP  routes 
and  observation  points  in  a  pre-mission  session.  To  objectively  assess  the  impact  of  the  moving  map 
capability  on  navigation,  the  positioning  of  the  ownship  relative  to  the  defined  flight  ingress  corridors 
leading  to  the  RP  was  measured.  The  impact  of  the  ERSTA-like  sensor  was  objectively  measured  as  the 
distance  at  which  the  aircrew  detected  and  identified  targets.  Subjective  ratings  of  performance  and  of 
task  difficulty  were  collected  after  each  mission.  Subjective  measures  of  navigation  and  mission 
performance  were  obtained  through  questionnaires  (see  Annex  B). 

It  was  hypothesized  that  the  digital  moving  map  and  the  ERSTA-like  sensor  would  affect  how  much  time 
the  MC  spent  looking  down  and  inside  the  cockpit.  This  was  objectively  assessed  by  recording  the  head 
positioning  of  the  MC  throughout  the  missions. 


Workload  -  Workload  was  assessed  subjectively  using  questionnaires  based  on  a  modified  NASA  TLX. 
Subjective  ratings  for  global  workload  were  obtained  as  were  ratings  for  specific  segments  (e.g.,  ingress, 
recce-zone,  egress)  in  the  missions.  Objective  measures  of  workload  were  not  directly  obtained. 

Situation  Awareness  -  The  objective  index  of  situation  awareness  was  the  percentage  of  objects  that 
aircrew  missed  during  each  mission.  After  each  mission,  pilots  also  rated  their  perceived  awareness 
overall  and  for  specific  information. 


1.3.  Predictions 

It  was  expected  that  relative  to  the  P-Map  condition,  the  presence  of  the  moving  map  Iin  the  M-Map  and 
M-Map/Sensor  conditions  would  increase  the  SA  of  the  MC,  especially  in  terms  of  spatial  and 
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navigational  SA.  It  was  also  anticipated  that  the  ERSTA-like  capabilities  would  generally  enhance  MC 
performance,  lower  the  difficulty  of  completing  tasks  for  the  MC,  and  lower  the  MC’s  workload.  The 
moving  map  was  also  predicted  to  increase  the  MC’s  confidence  in  re-creating  the  mission.  The  digital 
moving  map  and  sensor  capabilities  were  not  expected  to  significantly  affect  FP  ratings. 

2  Method 
2.1  Participants 

Participants  were  four  male  pilots  from  the  Canadian  Forces  ranging  in  age  from  26  to  40  years  (M  =  32). 
Years  in  the  Canadian  Forces  ranged  between  9  and  14.5  years  (M  =  12.125).  Years  as  pilots  ranged 
between  2  and  13  years  (M  =  6.5).  All  four  pilots  where  right  handed.  The  following  table  summaries 
estimated  time  in  hours  for  various  types  of  flights. 


Table  2.1  Estimated  Flight  Time 


Type  of  Flight 

Estimated  time  (in  hours) 

1.  Total  flight  time 

Range  750-2500  (M  =  1612.5) 

2.  Total  rotary  wing 

Range  500-2300  (M  =  1400) 

3.  Total  Griffon 

Range  400-900  (M  =  625) 

4.  Time  spent  using  a  flight  simulator 

Range  20-350  (M  =  147.5) 

All  four  participants  had  some  experience  with  flight  simulators.  These  included  full  motion  CH-146  as 
well  as  CF-18,  212  simulators,  Bell  212/412  and  AH-64A  Apache.  One  of  the  four  participants  had 
experienced  with  an  ERSTA  system  prior  to  this  experiment.  This  participant  served  as  an  ERSTA 
operator  (10-15  hours)  in  a  simulator  environment.  Two  of  the  four  participants  were  very  familiar  with 
the  Gagetown  NB  area,  and  thus  with  the  synthetic  terrain  environment  that  was  used  in  the  present 
experiment. 

2.2  Design 

This  experiment  compared  the  P-Map  versus  M-Map  versus  M-Map/Sensor  conditions.  The  objective 
and  subjective  measures  of  the  three  core  constructs  in  the  CSE  framework,  situation  awareness, 
workload,  and  task-relevant  performance  are  shown  in  Table  1.1. 


Table  2.2  CSE  Measures 


Type  of  Measure 

CSE  Domain 

Objective 

Subjective 

Task-Relevant 

Navigation  along  defined  flight 

Ratings  of  performance  and  task 

Performance 

corridors 

difficulty 

Distance  from  objective(s)/targets  for 
detection  &  identification 

Head  positioning  of  MC 

Situation  Awareness 

Detection  of  airborne  and  ground 

Ratings  of  SA  (specific  and 

objects 

global) 

Workload 

Not  obtained 

Ratings  of  workload 
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2.3  Materials  and  Measures 

Questionnaires  -  The  subjective  measurements  of  situation  awareness,  workload,  and  performance  were 
conducted  at  the  end  of  each  mission  and  at  the  end  of  the  experiment.  For  each  mission,  separate 
questionnaires  were  completed  by  the  FP  and  the  MC.  Participants  rated  variables  on  a  number  of  scales 
(refer  to  Annex  C).  In  addition,  before  starting  the  experiment,  participants  completed  a  background 
questionnaire  (see  Annex  D),  which  included  questions  about  the  number  of  tactical,  Griffon,  and 
simulator  flying  hours  they  had  logged. 

Head  Position  of  MC  -  Plead  position  data  was  collected  online  (30  hz)  to  determine  where  the  MC  was 
looking  throughout  each  mission.  From  this  data,  percent  head-up  versus  head-down  time  could  be 
calculated. 

Development  of  Mission  Scenarios  -  An  important  activity  in  this  experiment  was  to  develop 
representative  and  realistic  tactical  scenarios  that  could  be  used  for  both  for  this  experiment  and  as  a 
template  for  scenarios  in  the  follow  on  experiment  (Experiment  3)  in  the  TAMSS  SA  program.  This  goal 
was  accomplished  through  input  from  three  subject-matter  experts  (SMEs)  who  were  selected  to  provide 
different  backgrounds  and  perspectives.  One  SME  was  provided  by  DND.  This  SME  had  no  CEE  146 
Griffon  experience  but  he  was  a  highly  experienced  Kiowa  pilot  with  experience  in  tactical  helicopter 
operations.  The  second  SME  was  a  highly  experienced  Griffon  pilot  with  tactical  experience  and  with 
experience  at  DND  LATEF  facility  in  Gagetown  NB.  This  SME  had  recently  (<  10  months)  retired  from 
the  Canadian  Forces.  The  third  SME  was  a  retired  but  highly  experienced  fighter  jet  pilot.  This  SME  has 
a  vast  amount  of  experience  in  tactical  operations  involving  air  support. 

Numerous  entities  were  added  to  the  terrain  database,  creating  scenarios  that  would  allow  the 
experimenters  to  take  measures  of  Situation  Awareness,  Workload,  and  Task-relevant  performance.  The 
following  is  a  description  of  the  database  used  and  the  additions  made  to  the  external  scene  for  the 
purposes  of  the  experiment. 

Terrain  database  -  The  landscape  database  was  a  Virtual  Reality  model  of  a  40  km  (east  to  west)  by  50 
km  (north  to  south)  section  of  CEB  Gagetown,  NB,  divided  into  twenty  10  km  by  10  km  squares,  or 
“tiles”.  Due  to  computer  system  (memory)  constraints,  only  6  to  8  tiles  (i.e.,  the  minimum  required  to 
cover  the  terrain  relevant  to  a  given  mission)  were  displayed  during  any  one  mission.  The  database 
contained  a  number  of  fixed,  pre-determined  geographical  features  (river,  hills,  forest)  and  man-made 
elements  (barracks,  various  military  installations,  roads,  and  the  flight  base).  Various  entities,  both 
moving  and  stationary,  were  added  to  the  terrain  database  to  create  a  number  of  mission  scenarios  (see 
below)  in  order  to  assess  pilots’  situation  awareness  during  missions. 

The  objects  that  pilots  were  to  report  on  for  the  purpose  of  assessing  their  SA  were  inserted  and  controlled 
using  STAGE  software.  These  included: 

(a)  formation  of  five  Ml  13  armored  personnel  carriers  (mission  scenario  1), 

(b)  one  CEE146  Griffon  helicopter  flying  in  a  circle  pattern  (scenario  1), 

(c)  formation  of  BMP-3  light  infantry  vehicles  (scenario  1), 

(d)  formation  of  six  Ml  025  light  tactical  vehicles  (all  scenarios), 

(e)  formation  of  eight  M-109  artillery  vehicles  (scenarios  1  and  2), 

(f)  platoon  of  6  infantry  tents  (all  scenarios), 

(g)  formations  of  M2A3  Bradley  Fighting  Vehicles  (scenarios  1,2  and  4), 
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(h)  formation  of  six  M1A2  battle  tanks  (scenario  2), 

(i)  damaged  M939A2  5-ton  truck  (scenario  2), 

(j)  formation  of  four  BTR-80  armoured  personnel  carriers  (scenario  2), 

(k)  formation  of  six  Ml  10  artillery  vehicles  (scenario  3), 

(l)  crashed  Mi-28  attack  helicopter  (scenario  3), 

(m)  formation  of  six  M270  self-propelled  rocket  launchers  (scenario  3), 

(n)  formations  of  HUM  Avenger  armoured  vehicles  (scenarios  1,  3  and  4), 

(o)  formation  of  five  Ml  045  TOW  missile  carrier  vehicles, 

(p)  downed  Su-25  ground  attack  airplane,  and 

(q)  formation  of  four  ZSU-23-4  self-propelled  anti-aircraft  guns. 

The  vehicles  were  places  so  as  to  be  on  or  near  the  trajectories  or  Restricted  Operation  Zones  (ROZ’s). 
Most  entities  or  formations  were  stationary,  but  each  scenario  included  at  least  two  formations  of  moving 
vehicles  (these  tended  to  be  vehicle  formations  that  participants  had  not  been  briefed  on  at  the  outset  of 
their  missions).  All  entities  were  scaled  to  their  normal  size  relative  to  the  database. 

Mission  Scenarios  -  Four  mission  scenarios  were  developed,  with  terrain  features  and  vehicles  as 
described  above.  Descriptions  of  the  scenarios  are  included  in  Annex  A.  The  missions  were  roughly 
equivalent  in  complexity  and  number  of  entities,  and  all  followed  a  general  schema.  Participants  were 
first  briefed  on  the  general  context  of  the  mission,  an  operation  that  was  designed  to  resemble  United 
Nations  peace  support  operations  that  have  been  put  into  action  on  various  fronts  throughout  the  Balkans 
and  the  Middle  East  in  recent  years. 

Each  mission  consisted  of  a  starting  point  (which  also  served  as  endpoint),  a  release  point  (RP)  into  the 
Restricted  Operations  Zone  (ROZ),  a  point  through  which  the  ownship  was  supposed  to  exit  the  ROZ,  and 
an  intermediate  waypoint  connecting  both  the  RP  and  the  ROZ  to  the  start/endpoint  of  the  scenario.  The 
start/end  point,  intermediate  waypoint,  RP  and  ROZ  exit  point  were  all  connected  by  corridors  within 
which  the  aircrew  were  to  keep  the  ownship.  The  ownship’s  trajectory  within  the  ROZ  was  left  to  the 
discretion  of  the  pilots2. 

Each  mission  began  with  the  ownship  airborne  at  approximately  400  feet  above  ground  level  at  the  start 
point,  and  oriented  towards  the  first  waypoint.  The  waypoints  were  given  to  the  pilots  on  a  paper  map  and 
were  displayed  on  the  digital  moving  map,  but  the  terrain  database  itself  did  not  have  any  waypoint 
markers.  Participants  were  instructed  to  reach  the  RP  within  approximately  15  minutes  of  having  started 
the  mission,  and  were  given  a  maximum  of  30  minutes  to  observe  activity  in  the  ROZ.  They  were 
instructed  to  maintain  a  maximum  altitude  of  500  feet  while  transiting  from  the  start/end  point  and  the 
ROZ,  and  a  maximum  of  250  feet  in  the  ROZ  (the  ROZ  was  assumed  to  be  capped  by  active  high-speed 
airspace). 

In  each  mission,  participants  were  given  the  task  of  locating  and  assessing  the  state  of  an  objective 
(generally  a  bridge  or  some  other  strategic  landmark).  They  were  also  instructed  to  report  on  any  entities, 
air  or  ground,  they  detected  during  the  mission.  Participants  were  informed  in  advance  of  the  entities  they 
were  expected  to  encounter,  but  each  mission  included  three  entities  (one  enemy  formation,  one  friendly 
formation,  and  one  downed  vehicle)  that  were  not  briefed.  Also,  each  mission  included  unscripted 
weapons  activity  (directed  either  at  the  ownship  or  at  entities  in  the  terrain)  that  was  controlled  on-the-fly 
by  an  experimenter  using  the  STAGE  software.  The  unexpected  vehicles  and  the  unscripted  weapons 
activity  were  included  to  test  aircrew  SA  with  respect  to  unexpected  entities  and  events. 


2  The  ROZ  was  the  area  within  which  the  actual  reconnaissance  mission  was  supposed  to  take  place.  The 
participants  were  therefore  given  free  reign  to  determine  how  they  would  explore  it. 
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2.4  Procedure 

Upon  arrival  at  the  CACR  lab,  participants  were  provided  with  information  about  the  general  purpose  of 
the  experiment  and  were  given  an  overview  of  the  two-day  schedule.  Participants  then  completed  an 
informed  consent  and  the  personal  (experience)  background  questionnaire.  Following  the  information 
session,  the  participants  flew  a  practice  session  before  beginning  the  experimental  sessions.  For  the 
practice  session,  each  participant  flew  until  they  felt  comfortable  flying  tactical  with  the  simulator 
environment. 

Participants  were  run  in  pairs,  alternating  between  MC  versus  FP  roles  from  one  mission  to  the  next.  In 
total,  each  pair  took  part  in  six  missions,  with  each  participant  serving  as  the  MC  for  three  missions  and 
as  the  FP  for  three  missions.  Each  participant  served  as  the  MC  and  as  the  FP  in  the  Paper  Map,  Moving 
Map,  and  Moving  Map  +  Sensor  conditions. 

Testing  took  place  across  two  days.  Four  missions  were  flown  day  one:  two  missions  for  the  Paper  Map 
condition  and  two  missions  for  the  Moving  Map  condition.  On  the  second  day,  two  missions  were  flown 
for  the  Moving  Map  +  Sensor  condition. 

Two  experimenters  assisted  in  this  experiment.  One  experimenter  monitored  and  controlled  specific 
events  in  the  scenarios  using  STAGE.  The  second  experimenter  was  an  experienced  SME.  This  SME 
provided  radio  contact/communications  and  also  served  the  role  as  the  ERSTA-like  operator  in  the 
Moving  Map  +  Sensor  condition.  The  SME  communicated  with  the  pilots  via  the  simulator 
communications  system.  For  the  ERSTA  role,  the  SME  operated  an  ERSTA-like  station  that  was  situated 
behind  the  cockpit. 

Each  mission  took  approximately  20  minutes  to  complete.  After  each  mission,  the  participants  were  asked 
to  fill  out  the  mission  questionnaires.  At  the  end  of  the  second  day  the  participants  were  asked  to  fill  out  a 
final  questionnaire. 

3  Results 

The  experimental  goals  were  to  extend  the  evaluation  of  the  Cognitive  System  Engineering  (CSE) 
framework  proposed  by  the  C  ACR  for  evaluating  the  impact  of  novel  technology  on  aircrew  in  the 
CH146  Griffon  helicopter.  To  do  this,  the  impact  of  a  prototyped  ERSTA-like  system  was  assessed  with 
a  particular  focus  on  how  the  ERSTA-like  system  affects  the  performance,  situation  awareness  and 
workload  of  the  MC.  Three  conditions  were  examined:  P-Map  versus  M-Map  versus  M-Map/Sensor. 

Primary  activities  of  the  present  experiment  included  a)  integration  of  an  emulation  of  the  DND  CH-146 
ERSTA-like  system  into  the  CACR  simulator  using  HLA  protocol,  (b)  development  of  realistic  scenarios 
for  assessing  performance,  SA  and  workload  in  a  tactical  environment,  (c)  further  development  of  the 
data  collection  capabilities  of  the  CACR  simulation  facility  and  (d)  further  development  of  a 
questionnaire  battery  for  assessing  workload.  As  summarized  previously  in  Table  1.2,  objective  and 
subjective  measures  were  taken.  Because  of  the  small  sample  size  (four  participants),  only  descriptive 
statistics  are  reported  and  only  one  of  the  objective  measures  is  briefly  summarized. 

3.1  Objective  Measures 

Task-Relevant  Performance.  Head  position  data  for  the  MC  was  collected  throughout  each  mission. 
However,  technical  difficulties  are  such  that  stable  and  complete  data  was  only  obtained  for  one 
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participant.  This  data  for  this  participant  showed  that  percent  head-up  time  was  greater  in  the  M-Map 
(49%)  and  M-Map/Sensor  (48%)  conditions  than  in  the  P-Map  (33%)  condition.  This  suggests  that  the 
ERSTA-like  installation  had  the  positive  benefit  of  allowing  the  MC  to  spend  more  time  looking  outside 
the  cockpit.  Head-up  time  should  impact  on  flight  safety  and  enhance  the  contribution  of  the  MC  in 
detecting  and  responding  to  information  external  to  the  cockpit.  The  results  suggest  that  the  MC  was 
better  able  to  navigate  along  the  flight  corridors  when  the  moving  map  was  present  as  compared  to  the  P- 
Map  condition  where  only  a  hand  map  was  available.  In  addition,  it  is  clear  that  objectives  and  targets 
could  be  detected  and  identified  at  a  greater  distance  when  the  ERSTA-like  sensor  capability  was  present. 

Situation  Awareness  -  Aircrew  were  generally  proficient  at  detecting  the  relevant  airborne  and  ground 
objects,  regardless  of  which  condition  they  were  in.  However,  in  order  to  achieve  a  high  degree  of 
realism,  relative  few  objects  were  placed  in  the  experimental  scenarios.  To  obtain  a  more  sensitive 
objective  index  of  SA,  future  scenarios  will  need  to  include  more  objects,  some  of  which  would  be 
expected  and  others  that  would  be  unexpected. 

3.2  Post-Mission  Subjective  Ratings 

After  each  mission,  the  MC  and  the  FP  each  completed  subjective  questionnaires  in  which  they  rated  their 
performance,  difficulty  of  completing  tasks,  situation  awareness  and  workload.  The  responses  to  these 
questionnaires  are  given  separately  for  the  FP  versus  the  MC  position. 

Flying  Pilot  (FP).  The  subjective  ratings  for  the  FP  position  are  summarized  and  shown  in  Annex  E.  On 
average,  participants  rated  their  performance  in  the  FP  position  as  generally  better  in  the  M-Map  (4.5)  and 
M-Map/Sensor  (4.9)  condition  than  in  the  P-Map  (4.3)  condition.  These  values  all  fall  within  the  self- 
rated  categories  of  Adequate  to  Good  performance.  Relative  to  the  P-Map  condition,  the  M-Map  and  M- 
Map/Sensor  capabilities  were  judged  as  positively  affecting  most  aspects  of  navigation  and  positioning  of 
the  aircraft. 

Subjective  ratings  showed  that  regardless  of  condition,  difficulty  of  task  completion  was  usually  rated  as 
easy  to  moderate.  However,  many  tasks  were  judged  to  be  easier  in  the  M-Map  (2.7)  and  M-Map/Sensor 

(2.6)  conditions  than  in  the  P-Map  (3.35)  condition.  The  perceived  benefits  of  the  ERSTA-like 
capabilities  on  task  difficulty  were  especially  evident  for  finding  waypoints,  controlling  heading  and 
altitude.  As  anticipated,  there  was  no  impact  of  the  ERSTA-like  system  on  tasks  such  as  cross-checking 
instruments  and  controlling  airspeed. 

Ratings  of  situation  awareness  of  aircraft  systems  were  similar  across  the  P-Map  (4.57),  M-Map  (4.65) 
and  M-Map/Sensor  (4.9)  conditions.  These  values  refer  to  moderate  to  borderline  high  SA.  Similarly, 
awareness  of  mission-relevant  tactical  information  was  generally  undifferentiated  for  the  FP  role, 
although  rated  awareness  did  tend  to  increase  across  the  P-Map  (4.2),  M-Map  (4.4)  and  M-Map/Sensor 

(4.7)  conditions.  Average  ratings  of  spatial/nav  igational  awareness  showed  the  same  trend  from  P-Map 
(3.9),  M-Map  (4.2)  and  M-Map/Sensor  (4.4)  conditions.  Ratings  of  awareness  of  crew  activity  were 
similar  in  the  P-Map  (4.75),  M-Map  (4.5)  and  M-Map/Sensor  4.63)  conditions. 

On  average,  ratings  of  FP  workload  did  not  differ  across  the  P-Map  (3.0),  M-Map  (2.95)  and  M- 
Map/Sensor  (2.9)  conditions.  As  expected,  participants  did  note  that  FP  workload  was  higher  during 
enemy  engagement  and  when  hovering  the  simulator. 

Mission  Commander  (MC).  The  subjective  ratings  for  the  MC  position  are  summarized  and  shown  in 
Annex  F.  After  each  mission  the  participant  filling  the  MC  role  was  required  to  re-create  (draw)  the 
mission  using  the  paper  from  the  mission  map.  They  were  required  to  indicate  the  order  of 
waypoints/landmarks  visited,  showed  the  flight  path  with  lines  and  identify  the  type  and  location  of 
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activity  that  took  place  during  the  mission.  Following  this,  the  participant  rated  their  confidence  in  the  re¬ 
creation  of  the  mission.  The  average  rated  confidence  tended  to  increase  across  the  P-Map  (4.13)  to  the 
M-Map  (4.6)  and  M-Map/Sensor  (4.75)  conditions.  Participants  in  the  MC  position  noted  that  it  was 
easy  to  identify  terrain  with  the  SA  that  was  provided  by  the  of  the  moving  map  display,  but  that  some 
confusion  may  occur  if  there  were  any  differences  between  the  moving  map  and  the  paper  map.  It  was 
also  noted  that  it  was  difficult  to  re-create  a  mission  on  a  paper  map  when  the  mission  has  been  executed 
using  a  moving  map. 

Ratings  of  performance  for  various  tasks  in  the  MC  position  increased  on  average  from  “adequate”  in  the 
P-Map  (4.4)  condition  to  “good”  in  the  M-Map  (5.1)  and  M-Map/Sensor  (5.0)  conditions.  As  expected, 
performance  ratings  in  the  M-Map  and  M-Map/Sensor  conditions  were  noticeably  higher  than  the  P-Map 
condition  for  the  navigation  tasks  such  as  finding  waypoints,  reading  the  map  and  using  the  map  to 
navigate.  The  ERSTA-like  capabilities  were  also  rated  as  enhancing  the  positioning  of  the  aircraft  in  the 
recce  zone. 

For  the  MC  position,  average  rated  difficulty  of  performing  tasks  was  greater  in  the  P-Map  (4. 14)  than  in 
the  M-Map  (2.75)  and  M-Map/Sensor  (2.6)  conditions.  These  differences  were  especially  evident  for 
finding  waypoints,  reading  and  using  the  map  information,  route  planning  and  positioning  the  aircraft  in 
the  recce  zone.  Similarly,  the  difficulty  of  getting  information  during  a  mission  was  on  average  higher  P- 
Map  (3.71)  than  the  M-Map  (2.8)  and  M-map/Sensor  (2.45)  conditions. 

Ratings  of  situation  awareness  of  tactical  information  relevant  to  the  mission  were  generally  high  and 
undifferentiated  across  the  P-Map  (4.86)  versus  M-Map  (4.56)  conditions.  Rated  awareness  tended  to  be 
higher  in  the  M-Map/Sensor  (5.3 1)  condition.  The  SA  ratings  were  noticeably  higher  in  the  M- 
Map/Sensor  condition  for  tracking  the  unfolding  of  a  mission  and  for  anticipating  future  events.  Rating 
of  spatial/navigational  awareness  in  the  MC  position  also  increased  from  the  P-Map  (3.93)  to  the  M-Map 
(4.75)  and  M-Map/Sensor  (5.75)  conditions.  Importantly,  these  ratings  showed  a  clear  advantage  of  the 
M-Map/Sensor  condition  for  locating  ownship  relative  to  the  objective  (e.g.,  bridge)  and  relative  to 
enemy  activity  as  well  as  for  awareness  of  the  general  layout  of  the  navigated  area.  For  awareness  of 
crew  activities  while  in  the  MC  position,  average  ratings  did  not  differ  between  the  P-Map  (4.5)  and  M- 
Map  (4.63)  conditions,  but  were  generally  high  in  the  M-Map/Sensor  (5.5)  condition. 

Ratings  of  workload  during  a  mission  did  not  differ  dramatically  across  the  three  conditions,  but  on 
average  there  was  a  trend  toward  decreased  subjective  workload  from  the  P-Map  (3.95)  to  the  M-Map 
(3.30  and  M-Map/Sensor  (3.1)  conditions.  As  expected,  ratings  of  workload  were  highest  for  activity  in 
the  recce  zone  as  compared  to  the  Ingress  and  Egress  activities.  Written  comments  from  participants 
confirmed  that  workload  for  the  MC  was  high  in  the  recce  zone  “due  to  the  number  of  agencies  that 
needed  to  be  contacted  on  different  frequencies”.  It  was  also  noted  that  high  workload  for  the  MC  in  the 
recce  zone  was  mainly  associated  with  trying  to  maintain  SA  of  ownship  location.  One  piloted  noted  that 
the  moving  map  reduces  workload  related  to  navigation  thereby  freeing  more  time  for  msn  work  (comms, 
search  etc.). 

3.3  Post-Mission  Subjective  Ratings 

At  the  end  of  the  second  day  of  testing,  participants  completed  a  questionnaire  to  directly  compare  the 
three  experimental  conditions  for  both  the  FP  and  for  the  MC  roles.  The  average  ratings  are  shown  in 
Annex  G.  For  the  FP  position,  these  comparisons  show  that  relative  to  the  P-Map  condition  both  the  M- 
Map  and  the  M-Map/Sensor  conditions  were  perceived  as  improving  most  aspects  of  the  FP’s  SA, 
communications  between  the  FP  and  the  MC,  and  the  FP’s  performance.  Similar  enhancements  were 
reported  for  the  MC  position. 
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4.  Conclusions 

The  ability  to  generalize  from  this  experiment  is  limited  in  that  only  four  participants  were  run.  However, 
the  results  are  consistent  in  showing  perceived  advantages  for  the  ERSTA-like  digital  moving  map  and 
sensor  capabilities  over  the  current  Griffon  map  and  sensor  capabilities.  Participants  agreed  that  the 
moving  map  and  sensor  enhanced  the  MC’s  performance  and  SA  while  lowering  task  difficulty  and 
workload.  There  was  some  indication  of  these  benefits  being  transferred  to  the  FP,  particularly  in  terms 
of  the  aircrew’s  ability  to  position  the  aircraft  and  to  maintain  tactical  flight.  In  addition,  the  head 
tracking  data  showed  that  relative  to  the  P-Map  condition,  the  MC  was  able  to  spend  more  time  looking 
up  and  out  of  the  cockpit  in  the  M-Map  and  M-Map/Sensor  conditions. 

The  results  of  this  experiment  provide  a  solid  foundation  for  developing  the  third  and  final  experiment  in 
the  TAMSS  SA  project.  Of  importance  is  that  (a)  the  ERSTA-like  system  was  effectively  modeled  and 
integrated  into  the  simulator  environment  using  HLA  protocol,  (b)  the  scenarios  that  were  developed 
represented  realistic  tactical  missions,  (c)  the  questionnaire  battery  developed  for  obtaining  subjective 
measures  proved  to  be  sensitive  enough  to  index  and  differentiate  performance,  SA,  task  difficulty  and 
workload  across  the  three  experimental  conditions. 
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ANNEX  A 
Mission  Scenarios 
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Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  10  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN  troops 
out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this  territory  for  the  past  14 
months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive  them  out  of  the  territory.  Recently, 
in  the  past  four  months,  the  attacks  have  increased  in  number  and  in  ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD)  with  the 
main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to  live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having  been 
occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to  continue  the 
push  to  the  St  John  River.  These  key  areas  are  the  bridges  crossing  the  Oromocto,  Nerepis  and  Otnabog  Rivers  and 
choke  points  out  of  the  hilled  areas  on  the  West  side  of  the  St  John  river  leading  to  the  main  river  crossing  areas 
used  by  the  FDN  for  their  initial  invasion  some  14  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen  Stolitchnoya 
(callsign  Stiletto)  from  Lithuania.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out  of  the  territory. 
However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to  disrupt  the  UN  operation  and 
slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup  and  attempt  to  re-start  their  guerrilla 
activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  GE  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (Ml 09)  and  Armd  (Leo  2)  from  the  German  5  Spa-Panzer  Div  (SPD) 

•  UK  Inf  Bn  (3  Bn  Royal  Grenadiers) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AWACs  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (CF18s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  45  km  West  of  the  objective  area,  at  the  Brockway  Airfield  (FL 
480485). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These  include: 

•  bridges  at: 

.  GL  105558 

.  GL  097533 

.  GL  091499 

.  GL  063467 

.  GL  098450 

•  choke  points  at: 
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.  GL  187647 

.  GL  205578 

.  GL  190468 

•  GL  211431 

You  are  a  Griffon  crew  assigned  to  conduct  Recce  tasks  with  the  newly  equipped  ERST  A  system.  The  Comd 
UNMNB  has  been  impressed  with  the  support  provided  to  date  since  the  ERST  A  is  the  best  long  range  rapidly 
deployable  Recce  system  available  in  the  Bde.  The  ERSTA  equipped  Griffons  played  a  key  roll  in  aiding  the  rapid 
advance  of  the  MNB  in  the  past  9  days. 

It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant  importance.  The 
mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the  territory  may  be  halted  as  a 
result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  bridge  at  GL  091499.  The  last  report  from  the  patrol  indicated 
that  they  had  been  fired  at  sporadically  from  the  surrounding  hills  to  the  East  of  their  positions.  It  is  extremely 
important  that  the  HQ  determine  the  status  of  the  bridge  and  if  the  FDN  are  attempting  to  mine  or  destroy  the  bridge. 
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Pre  Fit  Brief  to  Crew  from  Ops 
Situation: 


General: 

•  as  per  intro 

En: 

•  likely  en  activity  in  area  of  GL  091499  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  GL  1856  -  6  hrs  ago 

•  intsum  -  bypassed  en  units  (Platoon  -)  East  of  20  Easting 

•  contact  report  -  en  patrol  raided  village  in  area  of  GL  0449  -  12  hrs  ago,  appeared  to  be 

headed  East 


Fr: 


Wx: 


Airspace: 

Mission: 


Recce  Tp  located  in  area  GL  0555  (6  x  HUM  TOW) 

Arty  Bty  located  at  FL  998504  (8  x  M109-A6) 

Inf  located  at  GL  059531  (camouflaged  tents) 

Patrol  on  bridge  at  GL  082519  (3  x  HUM  AVENGER  +  1  x  M2A3  BRADLEY) 
attached  for  duration  of  mission  -  W  Bty  through  G24 
CAS  on  call  from  MAGIC  (10  min.  notice  required) 


5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 
sunrise:  1200Z,  sunset:  0300Z 


restrictions  as  per  ACO 


Gain  observation  on  the  bridge  at  GL  091499  to  determine  if  the  bridge  is  intact  and  observe 
the  bridge  until  T43B  elements  arrive. 


Execution: 

General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

1 .  gain  observation  onto  the  bridge 

2.  determine  if  the  bridge  is  intact 

3.  maintain  observation  until  T43B  arrives 

Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  arty  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 


AD  Assets: 
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•  lx  M2A3  BRADLEY  is  located  at  GL  141579  (5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  AGO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 

Service  Support: 

•  DEWS  is  installed  and  configured  as  required 

.  FARP  at  GL  027481  (open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 

.  FSH  located  at  FL  884566 

•  pers  eqpt  -  SOP 

Command  and  Signals: 

Command: 

.  UC  408  Sqn  B  Fit 

.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  SPYGLASS 

•  objective  intact  MANHOLE 

•  objective  destroyed  COLD  START 

COMMs: 

•  as  per  CEOIs 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

TWEED 

FL  560565 

<500’ 

0930-1000 

BLISSVILLE 

FL  912532 

FINNEGANS 

FL  995577 

RP 

CLONES 

GL  057552 

<250’ 

N/A 

Egress 

PETERSVILLE 

GL  046498 

<500’ 

1000  -1100 

BLISSVILLE 

FL  912532 

TWEED 

FL  560565 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted 
from  other  friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The 
Restricted  Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  GL  106442 

.  GL  077450 

.  GL  095575 

.  GL  131563 

There  is  an  active  LLTR  immediately  South  of  the  objective  bridge,  passing  immediately  above  the  ROZ  (250’- 
500’),  one  way,  headed  SW.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the  LLTR  on  egress 
from  BAI  missions  in  FDN  held  territory  East  of  the  St  John  River.  The  LLTR  is  aligned  with: 

.  GL  2756  and  FL  9039 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AW  AC  (Magic).  Upon  passing  any  co-ord  point  or  release 
point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  FIR 

FL  850550,  FL  960613,  GL  040633,  FL  222655 
PL  OAK 

FL  850440,  GL  046498,  GL  256575 
PL  ASH 

FL  850330,  GL  078371,  GL  269496 
PL  CAT 

FL  956690,  FL  910623,  FL  897580,  FL  924465,  FL  992405,  FL  960330 
PL  DOG 

FL  965686,  FL  967640,  GL  006553,  GL  070435,  GL  087412,  GL  079369,  GL  1 17338 
PL  HORSE 

GL  000720,  GL  037662,  GL  059588,  GL  124550,  GL  1 14522,  GL  172502,  GL  247389 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AW  AC 

MAGIC 

HQII  /  A19.225 

No 

Bde  HQ 

92 

34.50 

Fill  5 

Patrol 

I23A 

33.90 

Fill  4 

Arty 

G24 

35.20 

Fill  3 

Recce  Tp 

T43B 

42.65 

Fill  1 

Fit  Ops 

0 

49.90 

Fill  2 

FARP 

52C 

46.50 

Fill  2 

CAS  (on  call) 

KUGAR 

HQII  /A03. 625 

No 

DEWS: 


NOMAD  Program  2 


IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

02 

0930-1000 

73 

1000-1030 

61 

1030-1100 

51 

1100-1130 

40 

1130-1200 

22 

Mode  2: 

N/A 

1324 

Mode  3 : 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination 
Point,  at  an  altitude  of 400 feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point. 
Once  the  aircrew  is  comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 

Experimenter  Inputs  (not  for  expert  user  consumption): 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of  the 
synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

•  5  x  Ml  13  APCs  will  transit  NW  on  the  black  track  from  GL  050553  to  GL  023569.  The  APCs  need  to  be 
moving  as  the  helicopter  is  transiting  between  FINNEGAN  and  CLONES. 

•  A  burning/crashed  AH-64  Apache  will  be  on  the  ground  at  GL  070554. 

•  4  x  BMP  Armoured  Fighting  Vehicles  will  transit  North  on  the  road  from  GL  092470  to  GL  1 14522.  The 
BMPs  need  to  be  moving  as  the  Griffon  reaches  CLONES. 
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Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  10  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN  troops 
out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this  territory  for  the  past  12 
months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive  them  out  of  the  territory.  Recently, 
in  the  past  three  months,  the  attacks  have  increased  in  number  and  in  ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD)  with  the 
main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to  live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having  been 
occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to  continue  the 
push  to  the  South  and  West  of  the  St  John  River.  These  key  areas  are  the  bridges  crossing  the  Nerepis  River  and 
choke  points  along  the  road  paralleling  the  NW-SE  rail  line  and  extending  South  to  the  St.  John  River.  Wellsford 
was  a  main  crossing  area  used  by  the  FDN  during  their  initial  incursion  some  12  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen  Ruberg 
(callsign  Thunder)  from  Lithuania.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out  of  the  territory. 
However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to  disrupt  the  UN  operation  and 
slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup  and  attempt  to  re-start  their  guerrilla 
activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  US  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (M109)  and  Armd  (ABRAMS)  from  the  1st  US  Armd  Div 

•  UK  Inf  Bn  (3  Bn  Royal  Grenadiers) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AW  AC  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (CF18s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  40  km  North  of  the  objective  area,  at  the  Hersey  Corner  Airstrip  (GL 
100792). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These  include: 

•  bridges  at: 

.  GL  091499 

.  GL  093466 

.  GL  099450 

.  GL  088433 

.  GL  089415 

•  choke  points  at: 

•  GL  079370 
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GL  017395 
FL  993403 
GL  226436 


It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant  importance.  The 
mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the  territory  may  be  halted  as  a 
result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  choke  point  at  GL  079370.  The  last  report  from  the  patrol 
indicated  that  they  had  observed  increased  vehicular  traffic  and  massing  of  Armd  unit(s)  to  the  South  of  their 
objective.  It  is  extremely  important  that  the  HQ  determine  the  status  of  the  choke  point  and  if  the  FDN  are 
attempting  to  advance  North. 
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Pre  Fit  Brief  to  Crew  from  Ops 
Situation: 


General: 

•  as  per  intro 

En: 

•  en  activity  in  area  of  GL  079370  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  GL  193431  -  3  hrs  ago 

•  contact  report  -  en  patrol  raided  village  in  area  of  GL  2243  -  8  hrs  ago,  appeared  to  be 
headed  West 


Fr: 


Wx: 


Airspace: 

Mission: 


Recce  Tp  located  in  area  GL  0345  (6  x  HUM  TOW) 
Arty  Bty  located  at  FL  050503  (8  x  M109-A6) 

Inf  Platoon  located  at  GL  107440  (camouflaged  tents) 
Patrol  on  bridge  at  GL  088433  (4  x  M2A3  BRADLEY) 
attached  for  duration  of  mission  -  W  Bty  through  G24 
CAS  on  call  from  MAGIC  (10  min.  notice  required) 


5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 
sunrise:  1200Z,  sunset:  0300Z 


restrictions  as  per  ACO 


Gain  observation  on  the  choke  point  at  GL  079370  (objective)  to  determine  en  activity  and 
continue  to  observe  until  T43B  elements  arrive. 


Execution: 

General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

4.  gain  observation  onto  the  choke  point 

5.  determine  if  there  is  any  en  advance  to  /  beyond  the  choke  point 

6.  maintain  observation  until  T43B  arrives 

Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  arty  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 


AD  Assets: 
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•  lx  M2A3  BRADLEY  is  located  at  FL  954438  (5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 

Service  Support: 

•  DEWS  is  installed  and  configured  as  required 

.  FARP  at  GL  141461(open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 

.  FSH  located  at  GL  098630 

•  pers  eqpt  -  SOP 

Command  and  Signals: 

Command: 

.  UC  408  Sqn  B  Fit 

.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  TELEPHOTO 

•  objective  secure  from  en  LOCKDOWN 

•  en  movement  North  of  objective  GALLOP 

COMMs: 

•  as  per  CEOIs 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

ROCKWELL  STREAM  BRIDGE 

GL  013734 

<500’ 

0930-1000 

KNOWLTON  HILL 

GL  060587 

BIG  BOG 

FL  997517 

RP 

RODDYS  LAKE 

GL  043443 

<250’ 

N/A 

Egress 

LYON  BRIDGE  ROAD 

GL  096451 

<500’ 

1000  -1100 

KNOWLTON  HILL 

GL  060587 

ROCKWELL  STREAM  BRIDGE 

GL  013734 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted 
from  other  friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The 
Restricted  Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  GL  074333 

.  FL  985393 

.  FL  976430 

.  GL  135366 

There  is  an  active  LLTR  immediately  West  of  the  objective  choke  point,  passing  immediately  above  the  ROZ  (250’- 
500’),  one  way,  headed  NNE.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the  LLTR  on  egress 
from  BAI  missions  in  FDN  held  territory  SW  of  the  St  John  River.  The  LLTR  is  aligned  with: 

.  FL  9926  and  FL  209778 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or  release 
point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  TROUT 

FL  890720,  FL  964686,  GL  107728,  GL  210779 
PL  PERCH 

FL  909501,  GL  060587,  GL  173569,  GL  253590 
PL  BASS 

FL  900320,  GL  017395,  GL  096451,  GL  260486 
PL  LION 

FL  931738,  FL  898560,  GL  010310 
PL  TIGER 

GL  237383,  GL  167607,  GL  138797 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AW  AC 

MAGIC 

HQII  /  A89.625 

No 

Bde  HQ 

92 

46.25 

Fill  5 

Patrol 

I22A 

39.80 

Fill  4 

Arty 

G14 

34.50 

Fill  3 

Recce  Tp 

T13B 

48.65 

Fill  2 

Fit  Ops 

0 

49.90 

Fill  1 

FARP 

52D 

46.50 

Fill  1 

CAS  (on  call) 

HAWK 

HQII /A67. 125 

No 

DEWS: 

NOMAD  Program  1 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

73 

0930-1000 

61 

1000-1030 

53 

1030-1100 

41 

1100-1130 

10 

1130-1200 

70 

Mode  2: 

N/A 

1324 

Mode  3 : 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination 
Point,  at  an  altitude  of  400  feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point. 

Once  the  aircrew  is  comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 

Experimenter  Inputs  (not  for  expert  user  consumption): 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of  the 
synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

•  6  x  M1A2  Abrams  MBTs  will  transit  WSW  on  the  road  from  GL  040497  to  FL  991481.  The  Abrams  need 
to  be  moving  as  the  helicopter  is  transiting  to,  but  short  of  BIG  BOG. 

•  A  burning  M939A2  5-Ton  Truck  will  be  on  the  ground  at  GL  067451. 

•  6  x  BTR  80s  will  transit  North  on  the  road  from  GL  100336  to  GL  079368.  The  BTRs  need  to  be  moving 
as  the  Griffon  reaches  RODDYS  LAKE. 


24 


TAMSS  SA  Experiment  2 


Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  10  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN  troops 
out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this  territory  for  the  past  10 
months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive  them  out  of  the  territory.  Recently, 
in  the  past  two  months,  the  attacks  have  increased  in  number  and  in  ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD)  with  the 
main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to  live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having  been 
occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to  continue  the 
push  to  the  North  and  West  of  the  St  John  River.  These  key  areas  are  the  bridges  West  of  Otnabog  Lake  and  choke 
points  on  the  West  side  of  the  St  John  river,  used  by  the  FDN  for  their  initial  incursion  some  10  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen  Stanlowski 
(callsign  Grimace)  from  Poland.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out  of  the  territory. 
However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to  disrupt  the  UN  operation  and 
slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup  and  attempt  to  re-start  their  guerrilla 
activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  GE  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (MLRS)  and  Armd  (Leo  2)  from  the  German  5  Spa-Panzer  Div  (SPD) 

.  UK  Arty  Bn  (21st  FA) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AW  AC  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (FI 8s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  35  km  South  West  of  Black  Clarendon  (FL  8521). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These  include: 

•  bridges  at: 


• 

GL 

170651 

• 

GL 

187647 

• 

GL 

196652 

• 

GL 

207649 

choke  points  at: 

.  GL  204704 

.  GL  174710 

.  GL  139640 
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GL  085735 
GL  041633 
GL  197657 


You  are  a  Griffon  crew  assigned  to  conduct  Recce  tasks  with  the  newly  equipped  ERST  A  system.  The  Comd 
UNMNB  has  been  impressed  with  the  support  provided  to  date  since  the  ERST  A  is  the  best  long  range  rapidly 
deployable  Recce  system  available  in  the  Bde.  The  ERSTA  equipped  Griffons  played  a  key  roll  in  aiding  the  rapid 
advance  of  the  MNB  in  the  past  7  days. 

It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant  importance.  The 
mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the  territory  may  be  halted  as  a 
result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  bridges  leading  to  the  choke  point  /  objective  area  at  GL  197657. 
The  last  report  from  the  patrol  indicated  that  they  had  taken  mortar  fire  at  sporadically  from  the  North  and  West  of 
their  positions.  It  is  extremely  important  that  the  HQ  determine  the  status  of  the  bridges  and  the  choke  point,  and  if 
the  FDN  are  attempting  to  mine  or  destroy  the  bridges  and  /or  advance  South  through  the  choke  point. 
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Pre  Fit  Brief  to  Crew  from  Ops 
Situation: 


General: 

•  as  per  intro 

En: 

•  likely  en  activity  in  area  of  GL  197657  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  GL  1774  -  8  hrs  ago 

•  intsum  -  ZSU-234  activity  at  Tantawanta  Bridge  -  4  hrs  ago 

•  contact  report  -  en  patrol  raided  village  in  area  of  Fentons  -  6  hrs  ago,  appeared  to  be 
headed  North  West 


•  Recce  Tp  located  in  area  GL  1472  (6  x  HUM  TOW) 

.  MLRS  Bty  located  at  GL  135585  (6  x  M270  MLRS) 

•  Inf  located  at  GL  210610  (camouflaged  tents) 

•  Patrol  at  road  intersection  GL  178629  (4  x  HUM  AVENGER) 

•  attached  for  duration  of  mission  -  W  Bty  through  G24 

•  CAS  on  call  from  MAGIC  (10  min.  notice  required) 

Wx: 

•  5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 

•  sunrise:  1200Z,  sunset:  0300Z 


Airspace: 

Mission: 


restrictions  as  per  ACO 


Gain  observation  on  the  key  bridges,  determine  if  intact  and  observe  the  choke  point  at  GL 
197657  until  T43B  elements  arrive. 


Execution: 

General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

7.  gain  observation  onto  the  key  bridges 

8.  determine  if  the  bridges  are  intact 

9.  maintain  observation  on  the  choke  point  until  T43B  arrives 

Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  MLRS  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 


AD  Assets: 
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•  lx  M2A3  BRADLEY  is  located  at  GL  249634(5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 

Service  Support: 

•  DEWS  is  installed  and  configured  as  required 

.  FARP  at  GL  068548  (open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 

.  FSH  located  at  GL  023485 

•  pers  eqpt  -  SOP 

Command  and  Signals: 

Command: 

.  UC  408  Sqn  B  Fit 

.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  WINDSCREEN 

•  all  bridges  intact  FANCY 

•  any  bridge  destroyed  BITTER 

•  en  movement  South  of  choke  point  LOCOMOTIVE 

COMMs: 

•  as  per  CEOIs 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

BLACK  CLAREDON 

FL  902340 

<500’ 

0930-1000 

WELLSFORD 

GL  078370 

BELL  BRIDGE  RUIN 

GL  097496 

RP 

DAY  HILL 

GL  193545 

<250’ 

N/A 

Egress 

MALLORY-KERR  ROADS 

GL  106578 

<500’ 

1000  -1100 

WELLSFORD 

GL  078370 

BLACK  CLAREDON 

FL  902340 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted 
from  other  friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The 
Restricted  Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  GL  120627 

.  GL  114683 

.  GL  232684 

.  GL  212622 

There  is  an  active  LLTR  immediately  West  of  the  objective  bridge,  passing  immediately  above  the  ROZ  (250’- 
500’),  one  way,  headed  SSE.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the  LLTR  on  egress 
from  BAI  missions  in  FDN  held  territory  North  of  the  St  John  River.  The  LLTR  is  aligned  with: 

.  GL  127820  and  GL  200320 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or  release 
point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  RAM 

FL  891789,  FL  992740,  GL  150734,  FL  262669 
PL  STEER 

FL  903627,  GL  023570,  GL  193502,  GL  262477 
PL  HOG 

FL  898419,  FL  947442,  GL  078370,  GL  230337 
PL  ROD 

FL  992740,  GL  041633,  GL  048500,  GL  000333 
PL  REEL 

GL  243792,  GL  262669,  GL  262447,  GL  230337 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AW  AC 

MAGIC 

HQII  /  A73.925 

No 

Bde  HQ 

92 

36.25 

Fill  5 

Patrol 

I22A 

59.80 

Fill  3 

Arty 

G14 

33.90 

Fill  4 

Recce  Tp 

T13B 

42.65 

Fill  1 

Fit  Ops 

0 

49.90 

Fill  2 

FARP 

52D 

46.50 

Fill  2 

CAS  (on  call) 

TIGER 

HQII  /  A48.625 

No 

DEWS: 

NOMAD  Program  1 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

03 

0930-1000 

51 

1000-1030 

73 

1030-1100 

41 

1100-1130 

30 

1130-1200 

00 

Mode  2: 

N/A 

1324 

Mode  3 : 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination 
Point,  at  an  altitude  of 400 feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point. 

Once  the  aircrew  is  comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 

Experimenter  Inputs  (not  for  expert  user  consumption): 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of  the 
synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

•  6  x  Ml  10  SP  Howitzers  will  transit  NNE  on  the  road  from  GL  154508  to  GL  169537.  The  Ml  10s  need  to 
be  moving  as  the  helicopter  is  transiting  between  BELL  BRIDGE  RUIN  and  DAY  HILL. 

•  A  landed  MI-28  Havoc  will  be  on  the  ground  at  GL  209629. 

•  4  x  T-72  MBTs  will  transit  South  on  the  road  from  GL  203703  to  the  choke  point  /  objective  at  GL  197657. 
The  MBTs  need  to  be  moving  as  the  Griffon  reaches  DAY  HILL. 
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Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  10  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN  troops 
out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this  territory  for  the  past  8 
months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive  them  out  of  the  territory.  Recently, 
in  the  past  month,  the  attacks  have  increased  in  number  and  in  ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD)  with  the 
main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to  live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having  been 
occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to  continue  the 
push  West  and  North  of  the  St  John  River.  These  key  areas  are  the  bridges  and  access  routes  crossing  the  Oromocto 
and  St.  John  (West  of  Gagetown)  Rivers  leading  to  the  main  incursion  points  used  by  the  FDN  for  their  initial 
invasion  some  8  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen  Leboeuf 
(callsign  Roaster)  from  France.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out  of  the  territory. 
However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to  disrupt  the  UN  operation  and 
slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup  and  attempt  to  re-start  their  guerrilla 
activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  GE  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (M109)  and  Armd  (Leo  2)  from  the  German  5  Spa-Panzer  Div  (SPD) 

.  UK  Arty  Bn  (21st  FA) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AW  AC  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (FI 8s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  40  km  South  East  of  the  objective  area,  at  the  Blue  Mountain 
Correctional  Facility  (GL  223453). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These  include: 

•  bridges  at: 

.  FL 904710 

.  FL  926797 

.  FL  929804 

.  FL  978822 

.  FL  960725 

.  FL 970777 

.  FL  996724 

•  choke  points  at: 
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.  FL 988780 

•  FL 978775 

•  FL 963752 

.  FL 996717 

You  are  a  Griffon  crew  assigned  to  conduct  Recce  tasks  with  the  newly  equipped  ERST  A  system.  The  Comd 
UNMNB  has  been  impressed  with  the  support  provided  to  date  since  the  ERST  A  is  the  best  long  range  rapidly 
deployable  Recce  system  available  in  the  Bde.  The  ERSTA  equipped  Griffons  played  a  key  roll  in  aiding  the  rapid 
advance  of  the  MNB  in  the  past  6  days. 

It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant  importance.  The 
mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the  territory  may  be  halted  as  a 
result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  bridge  at  FL  960725.  The  last  report  from  the  patrol  indicated 
that  they  had  been  fired  at  sporadically  from  the  buildings  to  the  South  West  of  their  positions.  It  is  extremely 
important  that  the  HQ  determine  the  status  of  the  bridge  and  if  the  FDN  are  attempting  to  mine  or  destroy  the  bridge. 
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Pre  Fit  Brief  to  Crew  from  Ops 
Situation: 


General: 

•  as  per  intro 

En: 

•  likely  en  activity  in  area  of  FL  960725  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  FL  8980  -  4  hrs  ago 

•  intsum  -  2S6  activity  in  the  area  of  Wood  Meadow  -  6  hrs  ago 

•  contact  report  -  en  patrol  raided  village  in  area  of  Lower  Lincoln  -  6  hrs  ago,  appeared  to 
be  headed  North  West 


Fr: 


Wx: 


Airspace: 

Mission: 


Recce  Tp  located  in  area  FL  9866  (6  x  F1UM  TOW) 

MLRS  Bty  located  at  FL  982652  (6  x  M270  MLRS) 

Inf  located  at  FL  994770  (camouflaged  tents) 

Patrol  on  bridge  at  FL  996724  (1  x  M2A3  BRADLEY  +  3  x  HUM  AVENGER) 
attached  for  duration  of  mission  -  W  Bty  through  G24 
CAS  on  call  from  MAGIC  (10  min.  notice  required) 


5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 
sunrise:  1200Z,  sunset:  0300Z 


restrictions  as  per  ACO 


Gain  observation  on  the  bridge  at  FL  960725  to  determine  if  the  bridge  is  intact  and  observe 
the  bridge  until  T43B  elements  arrive. 

Execution: 


General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

10.  gain  observation  onto  the  bridge 

1 1 .  determine  if  the  bridge  is  intact 

12.  maintain  observation  until  T43B  arrives 

Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  arty  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 


AD  Assets: 

•  lx  M2A3  BRADLEY  is  located  at  FL  960650  (5  km  Radius  down  to  50’) 
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Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 

Service  Support: 

•  DEWS  is  installed  and  configured  as  required 

.  FARP  at  GL  098722  (open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 

.  FSH  located  at  FL  195545 

•  pers  eqpt  -  SOP 

Command  and  Signals: 

Command: 

.  UC  408  Sqn  B  Fit 

.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  ZOOM 

•  objective  intact  HARVEST 

•  objective  destroyed  RECOIL 

COMMs: 

•  as  per  CEOIs 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

CENTRAL  HAMPSTEAD 

GL  250590 

<500’ 

0930  -1000 

L AWFIELD -B OUND ARY  ROADS 

GL  139640 

NW  KNOWLTON  HILL 

GL  048609 

RP 

BROAD  ROAD  CLEARING 

FL  984636 

<250’ 

N/A 

Egress 

LAUVINA  ROAD  CLEARING 

GL  006703 

<500’ 

1000  -1100 

L  AWFIELD  -B  OUND  ARY  ROADS 

GL  139640 

CENTRAL  HAMPSTEAD 

GL  250590 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted 
from  other  friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The 
Restricted  Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  FL  920678 

.  FL  936792 

.  FL  975782 

.  FL 973700 

.  FL  945672 

There  is  an  active  one  way  LLTR  South  of  the  objective  bridge,  headed  ENE  and  passing  immediately  South  of  the 
ROZ  (SFC  to  <  500’.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the  LLTR  on  egress  from  BAI 
missions  in  FDN  held  territory  West  of  the  objective.  The  LLTR  is  aligned  with: 

.  FL  8564  and  GL  2575 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or  release 
point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  SLEET 

FL  920832,  GL  094835,  GL  195794,  GL  262669 
PL  HAIL 

FL  901620,  GL  171502,  GL  249369 
PL  WAVE 

FL  960410,  FL  908510,  FL  897561,  FL  920832 
PL  RIPPLE 

GL  070305,  GL  097496,  GL  141579,  GL  139640,  GL  243793 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AW  AC 

MAGIC 

HQII/A1 1.125 

No 

Bde  HQ 

92 

37.85 

Fill  5 

Patrol 

I12A 

58.50 

Fill  1 

Arty 

G34 

34.50 

Fill  3 

Recce  Tp 

T21B 

30.65 

Fill  2 

Fit  Ops 

0 

49.90 

Fill  4 

FARP 

52S 

46.50 

Fill  4 

CAS  (on  call) 

STING 

HQII  /  A55.525 

No 

DEWS: 

NOMAD  Program  4 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

13 

0930-1000 

41 

1000-1030 

23 

1030-1100 

61 

1100-1130 

70 

1130-1200 

21 

Mode  2: 

N/A 

1721 

Mode  3 : 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination 
Point,  at  an  altitude  of 400 feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point. 

Once  the  aircrew  is  comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 

Experimenter  Inputs  (not  for  expert  user  consumption): 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of  the 
synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

.  5  x  Ml 045  HHMWV  TOWs  will  transit  NW  on  the  black  track  from  FL  983614  to  FL  968640.  The  TOWs 

need  to  be  moving  as  the  helicopter  is  transiting  NW  KNOWLTON  FULL. 

•  A  burning/crashed  SU-25  Frogfoot  will  be  on  the  ground  at  FL  930697. 

•  4  x  ZSU-234s  will  transit  South  on  the  road  from  FL  965765  to  FL  963746.  The  ZSU-234s  need  to  be 
moving  as  the  Griffon  reaches  BROAD  ROAD  CLEARING. 
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ANNEX  B 

Mission  Assessment  Questionnaires 
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TAMSS  Experiment  2 
Mission  Assessment 


Participant  ID 

Date 

Time 

Mission  number 

Scenario  Code 

Condition 

Hand  Map  Or  Moving  Map  OR  Moving 

Map  + 

Sensor 

Role 

Mission  Commander  Or  Flying  Pilot 

Instructions 

If  you  have  additional  comments  about  any  question  or  item,  please  write  these  on 
the  back  of  the  corresponding  page. 
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Ml.  Rate  how  well  you  performed  of  each  of  the  following  tasks  during  this  mission. 
Circle  N/A  if  the  item  was  not  applicable  to  this  mission. 


PERFORMANCE 

Task 

very  very 

poor  adequate  good 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Cross  checking  relevant 

instruments/ symbology 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  information  from 

the  external  scene  to 

control  the  aircraft 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  aircraft  in 

recce  area 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  tactical 

flight 

1 

2 

3 

4 

5 

6 

7 

NA 
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M2.  Rate  the  difficulty  of  each  of  the  following  tasks  during  the  mission.  Circle  N/A  if 
the  item  was  not  applicable  to  this  mission. 


DIFFICULTY 

Task 

very  very 

easy  moderate  difficult 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Cross  checking  relevant 

instruments/symbology 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  information  from 

the  external  scene  to 

control  the  aircraft 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  aircraft  in 

recce  area 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  tactical 

flight 

1 

2 

3 

4 

5 

6 

7 

NA 
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M3.  Rate  your  awareness  of  the  status  of  the  aircraft  systems  as  it  applies  to  your 
mission/task. 


AWARENESS 

very  moderate  very 

low  high 

Aircraft  Systems 

overall 

1 

2 

3 

4 

5 

6 

7 

NA 

Heading 

1 

2 

3 

4 

5 

6 

7 

NA 

RAD  Alt 

1 

2 

3 

4 

5 

6 

7 

NA 

Airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Attitude 

1 

2 

3 

4 

5 

6 

7 

NA 

1 

2 

3 

4 

5 

6 

7 

NA 

M4.  Rate  your  awareness  of  tactical  information  relevant  to  your  mission  (i.e.  where  do  you 
need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the  mission 
unfolding). 


AWARENESS 

very  moderate  very 

low  high 

Overview  of  mission 

1 

2 

3 

4 

5 

6 

7 

NA 

Unfolding  of 

mission/keeping  track 

of  how  mission 

unfolds 

1 

2 

3 

4 

5 

6 

7 

NA 

Potential 

developments 

(anticipating  future 

scenarios) 

1 

2 

3 

4 

5 

6 

7 

NA 
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Global  mission  goals 

1 

2 

3 

4 

5 

6 

7 

NA 

Specific  mission  goals 

1 

2 

3 

4 

5 

6 

7 

NA 

Enemy  activities 

1 

2 

3 

4 

5 

6 

7 

NA 

Friendly  activities 

1 

2 

3 

4 

5 

6 

7 

NA 

General  threat 

1 

2 

3 

4 

5 

6 

7 

NA 

Where  I  need  to  go 

1 

2 

3 

4 

5 

6 

7 

NA 

M5.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location  of 
ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects  and  landmarks  in 
the  environment). 


AWARENESS 

very  moderate  very 

low  high 

Overall  Spatial 

Orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  target  (e.g. 

bridge) 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  enemy 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  friendly 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Target  location 

relative  to  enemy  and 

friendly  units 

1 

2 

3 

4 

5 

6 

7 

NA 

Important  landmarks 

1 

2 

3 

4 

5 

6 

7 

NA 

General  layout  of  the 

navigated  area 

1 

2 

3 

4 

5 

6 

7 

NA 
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M6.  Rate  the  following  crew  activity. 


very  moderate  very 

low  high 

Overall  quality  of 

communication 

1 

2 

3 

4 

5 

6 

7 

NA 

The  usefulness  of 

information  provided 

by  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

NA 
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M7.  Rate  your  workload  during  this  mission.  Circle  NA  if  not  applicable  during  this  mission. 


WORKLOAD 

very  moderate  very 

low  high 

Ingress  to  first 

waypoint 

1 

2 

3 

4 

5 

6 

7 

NA 

First  waypoint  to 

release  point  (RP) 

1 

2 

3 

4 

5 

6 

7 

NA 

Recce  zone  [release 

point  to  target] 

1 

2 

3 

4 

5 

6 

7 

NA 

Egress  [target  to  end] 

1 

2 

3 

4 

5 

6 

7 

NA 

Overall 

1 

2 

3 

4 

5 

6 

7 

NA 

At  any  point,  did  you  find  that  your  workload  was  very  different  (e.g.,  much  higher  or  lower) 
than  across  the  mission  as  a  whole?  YES  or  NO 
If  yes,  please  elaborate: 
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TLX  WORKLOAD  ASSESSMENT 

Instructions.  Place  an  X  on  each  scale  at  the  point  that  represents  the  magnitude  of  each  factor  in 
the  mission  you  just  performed.  Refer  to  the  Workload  Scale  Descriptions  for  definitions  of  each 
factor. 


SEGMENT  OF  MISSION:  Ingress  to  first  waypoint 


Mental  demand  LOW 

Physical  demand  LOW 

Temporal  demand  LOW 

Frustration  level  LOW 

Effort  LOW 

Performance  POOR 


HIGH 

HIGH 

HIGH 

HIGH 

HIGH 

GOOD 


SEGMENT  OF  MISSION:  First  waypoint  to  release  point 


Mental  demand  LOW 

Physical  demand  LOW 

Temporal  demand  LOW 

Frustration  level  LOW 

Effort  LOW 

Performance  POOR 


HIGH 

HIGH 

HIGH 

HIGH 

HIGH 

GOOD 
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SEGMENT  OF  MISSION:  Recce  zone  [release  point  to  target] 

Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  | — | — | — | — | — | — | — | — | — | — |  GOOD 

SEGMENT  OF  MISSION:  Egress  [target  to  END] 

Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  I — I — I — I — I — I — I — I — I — I — I  GOOD 
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ANNEX  C 

Post  Mission  Questionnaires 
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Summary  of  Post  Mission  Questionnaires 
TAMSS  Project  -  Experiment  2 

Flying  Pilot 


Rated  Performance 

Ml .  Rate  how  well  you  perfonned  of  each  of  the  following  tasks  during  this  mission.  Circle 
N/A  if  the  item  was  not  applicable  to  this  mission. 

Average  scores;  poor  (1-3),  adequate  (4)  and  good  (5-7) 

Average  score  for  rated  performance  overall;  hand  map  alone  =  4.3,  moving  map  =  4.5, 
moving  map  and  sensor  =  4.9 


Hand  map  only 

Moving  map 

Moving  map  and 
sensor 

Finding  waypoints 

Good  (4.5) 

Good  (4.33) 

Good  (5) 

Control  heading 

Good  (4.5) 

Good  (4.75) 

Good  (5) 

Control  altitude 

Adequate  (4) 

Good  (4.5) 

Good  (4.75) 

Control  airspeed 

Adequate  (4) 

Adequate  (4) 

Good  (4.5) 

Cross  checking 
relevant 

instrument/symbology 

Good  (4.5) 

Good  (4.5) 

Good  (5) 

Using  information 
from  the  external 
scene  to  control  the 
aircraft 

Good  (4.5) 

Good  (4.5) 

Good  (4.75) 

Positioning  the 
aircraft  in  recce  area 

Adequate  (4.25) 

Good  (4.75) 

Good  (5.25) 

Maintaining  tactical 
flight 

Good  (4.5) 

Good  (4.75) 

Good  (5.25) 
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Rated  Task  Difficulty 

M2.  Rate  the  difficulty  of  each  of  the  following  tasks  during  the  mission.  Circle  N/A  if  the  item 
was  not  applicable  to  this  mission. 

Average  scores;  easy  (1-3),  moderate  (4)  and  difficult  (5-7) 

Average  score  for  rated  difficulty  overall;  hand  map  alone  =  3.35,  moving  map  =  2.7, 
moving  map  and  sensor  =  2.6 


Hand  map  only 

Moving  map 

Moving  map  and 
sensor 

Finding  waypoints 

Difficult  (4.33) 

Easy  (2.5) 

Easy  (2.67) 

Control  heading 

Easy  (3.75) 

Easy  (2.5) 

Easy  (2.25) 

Control  altitude 

Moderate  (4) 

Easy  (2.5) 

Easy  (2.25) 

Control  airspeed 

Easy  (2.5) 

Easy  (3) 

Easy  (2.75) 

Cross  checking 
relevant 

instrument/symbology 

Easy  (2.5) 

Easy  (2.75) 

Easy  (2.75) 

Using  information 
from  the  external 
scene  to  control  the 
aircraft 

Easy  (3.5) 

Easy  (2.75) 

Easy  (2.5) 

Positioning  the 
aircraft  in  recce  area 

Easy  (3.25) 

Easy  (3) 

Easy  (2.5) 

Maintaining  tactical 
flight 

Easy  (3) 

Easy  (2.75) 

Easy  (3) 
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Rated  Situation  Awareness 

M3.  Rate  your  awareness  of  the  status  of  the  aircraft  systems  as  it  applies  to  your  mission/task. 
Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  score  for  rated  awareness  of  system’s  status  overall;  hand  map  alone  =  4.57, 
moving  map  =  4.65,  moving  map  and  sensor  =  4.9 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Aircraft  systems 
overall 

Moderate  (3.33) 

Moderate  (4.25) 

Moderate  (4.25) 

Heading 

High  (4.75) 

High  (4.5) 

High  (5) 

RAD  alt 

High  (5) 

High  (5) 

High  (5.5) 

Airspeed 

High  (5) 

High  (4.75) 

High  (5) 

Attitude 

High  (4.75) 

High  (4.75) 

High  (4.75) 
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M4.  Rate  your  awareness  of  tactical  infonnation  relevant  to  your  mission  (i.e.  where  do  you 
need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the  mission 
unfolding). 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  score  for  rated  awareness  of  mission  overall;  hand  map  alone  =  4.2,  moving  map  = 
4.4,  moving  map  and  sensor  =  4.7 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Overview  of 
mission 

High  (4.75) 

High  (4.5) 

High  (5) 

Unfolding  of 
mission/keeping 
track  of  how 
mission  unfolds 

Moderate  (4) 

High  (4.75) 

High  (4.5) 

Potential 
developments 
(anticipating  future 
scenarios) 

High  (4.5) 

Low  (3.75) 

High  (4.5) 

Global  mission 
goals 

High  (4.5) 

High  (4.5) 

High  (4.75) 

Specific  mission 
goals 

High  (5) 

High  (4.75) 

High  (4.75) 

Enemy  activities 

Low  (3.5) 

High  (4.5) 

High  (5) 

Friendly  activities 

Low  (3.75) 

Low  (3.75) 

Moderate  (4.5) 

General  threat 

Moderate  (4.25) 

Moderate  (4.25) 

High  (4.5) 

Where  I  need  to  go 

Low  (3.75) 

High  (5) 

High  (4.75) 
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M5.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location  of 
ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects  and 
landmarks  in  the  environment). 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  score  for  rated  awareness  of  mission  overall;  hand  map  alone  =  3.9,  moving  map  = 
4.2,  moving  map  and  sensor  =  4.4 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Overall  spatial 
orientation 

High  (4.5) 

Moderate  (4.25) 

High  (4.75) 

Ownship  location  in 
relation  to  target 
(e.g.  bridge) 

Moderate  (4.25) 

High  (4.5) 

High  (4.5) 

Ownship  location  in 
relation  to  enemy 
activity 

Moderate  (4) 

Moderate  (4.25) 

High  (4.5) 

Ownship  location  in 
relation  to  friendly 
activity 

Low  (3.75) 

Low  (3.5) 

Low  (3.75) 

Target  location 
relative  to  enemy 
and  friendly  units 

Low  (3.75) 

Moderate  (4.25) 

Moderate  (4) 

Important 

landmarks 

Low  (3.25) 

Moderate  (4.25) 

High  (4.5) 

General  layout  of 
the  navigated  area 

Low  (3.75) 

High  (4.5) 

High  (4.5) 

M6.  Rate  the  following  crew  activity. 

Average  score  for  rated  awareness  of  crew  activity  overall ;  hand  map  alone  =  4.75, 
moving  map  =  4.5,  moving  map  and  sensor  =  4.625 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Overall  quality  of 
communication 

High  (4.5) 

Moderate  (4.25) 

Moderate  (4.25) 

The  usefulness  of 
information 
provided  by  Mission 
Commander 

High  (5) 

High  (4.75) 

High  (5) 
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Rated  Workload 

M7.  Rate  your  workload  during  this  mission.  Circle  NA  if  not  applicable  for  this  mission. 

Average  score  for  rated  workload  overall;  hand  map  alone  =  3,  moving  map  =  2.95, 
moving  map  and  sensor  =  2.9 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Ingress  to  first 
waypoint 

Low  (2.25) 

Low  (2) 

Low  (2) 

First  waypoint  to 
release  point 

Low  (2.75) 

Low  (2.75) 

Low  (2) 

Recce  zone  [release 
point  to  target] 

Moderate  (4) 

Low  (3.5) 

Moderate  (4.25) 

Egress  [target  to 
end] 

Low  (3) 

Low  (2.5) 

Low  (3) 

Overall 

Low  (3.25) 

Moderate  (4) 

Low  (3.25) 

Additional  question:  At  any  point  did  you  find  that  your  workload  was  very  different  (e.g.  much 
higher  or  lower)  than  across  the  mission  as  a  whole? 

PI :  During  enemy  engagement  in  vicinity  of  objectives 
P3:  Higher  workload  when  hovering  the  simulator 
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Summary  of  Post  Mission  Questionnaires 
TAMSS  Project  -  Experiment  2 


Mission  Commander 


Rated  Confidence  in  Re-creating  Mission 

Ml .  Please  re-create  the  mission  you  flew  using  your  mission  map.  Using  numbers,  indicate  the 
orders  of  waypoints/landmarks  visited:  show  the  flight  path  with  lines.  Identify  the  activity 
that  takes  place  during  the  mission.  Indicate  where  it  takes  place.  After  you  have  finished 
re-creating  this  mission,  rate  your  confidence  in  the  accuracy  of  your  re-creation  of  this 
mission. 

Average  rated  confidence  overall;  hand  map  alone  =  4.13,  moving  map  =  4.6,  moving  map 
and  sensor  =  4.75 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Re-creating  the 
flight  path  in  the 
recce  zone 

High  (4.75) 

High  (5) 

High  (5) 

Identifying  activity 
along  the  flight  path 
IN  the  recce  zone 

Low  (3.75) 

Moderate  (4.25) 

High  (4.75) 

Identifying  activity 
along  the  flight  path 
before  the  recce 

zone 

Low  (4) 

High  (4.5) 

High  (4.5) 

Overall 

Moderate  (4) 

High  (4.5) 

High  (4.75) 

Participants’  additional  comments 

PI :  When  manoeuvring  under  enemy  fire  it  is  difficult  to  re-create  the  path. 

PI :  Definitely  more  difficult  to  re-create  mission  on  a  paper  map  when  mission  was  executed 
using  a  digital  map. 

P2:  Very  difficult  to  identify  activity  without  sensors. 

P4:  Very  easy  to  identify  terrain  with  the  SA  provided  by  the  moving  map  display.  Differences 
between  moving  map  and  paper  map  may  cause  some  confusion. 
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Rated  Performance 

M2.  Rate  how  well  you  performed  of  each  of  the  following  tasks  during  this  mission.  Circle 
N/A  if  the  item  was  not  applicable  to  this  mission. 

Average  scores;  poor  (1-3),  adequate  (4)  and  good  (5-7) 

Average  rated  performance  overall;  hand  map  alone  =  4.4,  moving  map  =  5.1,  moving  map 
and  sensor  =5.0 


Hand  map  only 

Moving  map 

Moving  map  and 
sensor 

Finding  waypoints 

Poor  (3.33) 

Good  (5.25) 

Good  (5.5) 

Using  the  CDU 

Good  (4.67) 

Good  (4.75) 

Good  (5.25) 

Reading  the  map 

Adequate  (4.25) 

Good  (5.25) 

Good  (5) 

Using  the  map  to 
navigate 

Good  (4.5) 

Good  (5.5) 

Good  (5.25) 

Planning  the  route 

Good  (4.75) 

Good  (5) 

Good  (4.75) 

Using  comms 

Good  (4.75) 

Adequate  (4.25) 

Good  (4.5) 

Communicating  with 
the  ERSTA  operator 

Good  (5.25) 

Guiding  the  ERSTA 
operator  to  position 
the  sensor 

Good  (4.5) 

Guiding  the  flying 
pilot  to  position  the 
aircraft  in  recce  zone 

Good  (4.5) 

Good  (5.67) 

Good  (5) 
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Rated  Task  Difficulty 

M3.  Rate  the  difficulty  of  each  of  the  following  tasks  during  the  mission.  Circle  N/A  if  the  item 
was  not  applicable  to  this  mission. 

Average  scores;  easy  (1-3),  moderate  (4)  and  difficult  (5-7) 

Average  rated  difficulty  overall;  hand  map  alone  =  4. 14,  moving  map  =  2.75,  moving  map 
and  sensor  =  2.6 


Hand  map  only 

Moving  map 

Moving  map  and 
sensor 

Finding  waypoints 

Difficult  (5.33) 

Easy  (2.75) 

Easy  (2) 

Using  the  CDU 

Easy  (3.67) 

Easy  (2.75) 

Easy  (2.5) 

Reading  the  map 

Difficult  (4.5) 

Easy  (2.75) 

Easy  (2.5) 

Using  the  map  to 
navigate 

Moderate  (4.25) 

Easy  (2.75) 

Easy  (2.75) 

Planning  the  route 

Moderate  (4) 

Easy  (2.75) 

Easy  (2.5) 

Using  comms 

Easy  (3) 

Easy  (3.5) 

Easy  (2.75) 

Communicating  with 
the  ERSTA  operator 

Easy  (2.5) 

Guiding  the  ERSTA 
operator  to  position 
the  sensor 

Easy  (2.75) 

Guiding  the  flying 
pilot  to  position  the 
aircraft  in  recce  zone 

Moderate  (4.25) 

Easy  (2) 

Easy  (2.75) 
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M4.  Rate  the  difficulty  of  getting  information  from  each  of  the  following  sources  during  this 
mission.  Circle  NA  if  not  applicable  to  this  mission. 

Average  scores;  easy  (1-3),  moderate  (4)  and  difficult  (5-7) 

Overall  rated  difficulty  of  getting  information;  hand  map  alone  =  3.71,  moving  map  =  2.8, 
moving  map  and  sensor  =  2.45 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Hand  map 

Easy  (3.75) 

Easy  (3) 

Easy  (2) 

Moving  map 

Easy  (2.5) 

Easy  (2.5) 

ERSTA  sensor 

Easy  (2.5) 

ERSTA  operator 

Easy  (2.25) 

CDU 

Easy  (3.67) 

Easy  (3) 

Easy  (3) 

59 


TAMSS  SA  Experiment  2 


Rated  Situation  Awareness 

M5.  Rate  your  awareness  of  tactical  infonnation  relevant  to  your  mission  (i.e.  where  do  you 
need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the  mission 
unfolding). 

Average  scores;  poor  (1-3),  moderate  (4)  and  high  (5-7) 

Overall  rated  awareness  of  mission;  hand  map  alone  =  4.86,  moving  map  =  4.56,  moving 
map  and  sensor  =  5.31 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Overview  of 
mission 

High  (5.25) 

High  (4.75) 

High  (5.25) 

Unfolding  of 
mission/keeping 
track  of  how 
mission  unfolds 

High  (4.75) 

High  (4.75) 

High  (5.75) 

Potential 
developments 
(anticipating  future 
scenarios) 

High  (4.75) 

High  (4.5) 

High  (5.75) 

Global  mission 
goals 

High  (4.5) 

Moderate  (4.25) 

High  (5) 

Specific  mission 
goals 

High  (5) 

High  (5) 

High  (5.5) 

Enemy  activities 

High  (4.5) 

Moderate  (4) 

High  (5) 

Friendly  activities 

High  (4.5) 

Moderate  (4.25) 

High  (5.25) 

General  threat 

High  (5.25) 

Moderate  (4.25) 

High  (4.75) 

Where  I  need  to  go 

High  (5.25) 

High  (5.25) 

High  (5.5) 
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M6.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location  of 
ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects  and 
landmarks  in  the  environment). 

Average  scores;  poor  (1-3),  moderate  (4)  and  high  (5-7) 

Overall  rated  spatial  awareness;  hand  map  alone  =  3.93,  moving  map  =  4.75,  moving  map 
and  sensor  =  5.75 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Overall  spatial 
orientation 

Moderate  (4) 

High  (5.25) 

High  (5.5) 

Ownship  location  in 
relation  to  target 
(e.g.  bridge) 

High  (4.75) 

High  (5.25) 

High  (6) 

Ownship  location  in 
relation  to  enemy 
activity 

Low  (3.5) 

Moderate  (4.25) 

High  (5.5) 

Ownship  location  in 
relation  to  friendly 
activity 

Moderate  (4) 

Moderate  (4) 

High  (6) 

Target  location 
relative  to  enemy 
and  friendly  units 

Moderate  (4) 

High  (4.5) 

High  (5.75) 

Important 

landmarks 

Low  (3.5) 

High  (4.75) 

High  (5.5) 

General  layout  of 
the  navigated  area 

Low  (3.75) 

High  (5.25) 

High  (6) 
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M7.  Rate  the  following  crew  activity. 

Average  scores;  poor  (1-3),  moderate  (4)  and  high  (5-7) 

Overall  rated  crew  awareness/activity;  hand  map  alone  =  4.5,  moving  map  =  4.625,  moving 
map  and  sensor  =  5.5 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Overall  quality  of 
communication 

Moderate  (4.25) 

Moderate  (4.25) 

High  (5.5) 

The  usefulness  of 
information 
provided  by  ERSTA 
officer 

High  (5.75) 

Ability  to  instruct 
ERSTA  operator 

High  (5.25) 

Awareness  of 

ERSTA  operator’s 
activity 

High  (5.75) 

Ability  to  convey 
information  to 
flying  pilot 

High  (4.75) 

High  (5) 

High  (5.25) 
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Rated  Workload 

M7.  Rate  your  workload  during  this  mission.  Circle  NA  if  not  applicable  for  this  mission. 

Average  rated  workload  overall;  hand  map  alone  =  3.95,  moving  map  =  3.3,  moving  map 
and  sensor  =  3.1 


Hand  map  alone 

Moving  map 

Moving  map  and 
sensor 

Ingress  to  first 
waypoint 

Low  (3.5) 

Low  (2.5) 

Low  (2) 

First  waypoint  to 
release  point 

Moderate  (4.25) 

Low  (2.75) 

Low  (2.5) 

Recce  zone  [release 
point  to  target] 

High  (4.75) 

Low  (3.75) 

Moderate  (4) 

Egress  [target  to 
end] 

Low  (3) 

Moderate  (4) 

Low  (3.67) 

Overall 

Moderate  (4.25) 

Low  (3.5) 

Low  (3.5) 

Additional  question:  At  any  point  did  you  find  that  your  workload  was  very  different  (e.g.  much 

higher  or  lower)  than  across  the  mission  as  a  whole? 

Paper  map  condition:  YES(4)  NO(O) 

P 1 :  During  the  actual  recce  due  to  the  number  of  agencies  needed  to  be  contacted  on  different 
frequencies. 

P2:  High  workload  in  recce  zone  -  mainly  associated  with  trying  to  maintain  SA  of  own 
location. 

Moving  map  condition:  YES(2)  NO(2) 

PI :  Workload  is  higher  in  the  recce  zone. 

P4:  During  observation  of  the  OBJ,  the  workload  for  the  MC  is  more  demanding  because  of  the 
requirements  for  continuous  communications  with  various  call  signs 

Moving  map/sensor  condition:  YES(2)  NO(2) 

P2:  Busier  at  OPs  because  able  to  see  more.  Moving  map  reduces  workload  related  to 
navigation  freeing  more  time  for  msn  work  (coimns,  search  etc.) 

P4:  During  the  engagement  period  of  the  EN  vehicles.  The  COMMS  was  demanding  causing 
heavier  workload  the  rest  of  the  mission. 
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ANNEX  D 

Background  Questionnaire 
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TAMSS  Experiment  2  Participant  ID: _ 

Background  Information 

This  information  helps  us  to  determine  whether  we  have  recruited  a  broad  sample  of  pilots.  To 
get  an  unbiased  understanding  of  the  concerns  of  the  operation  community,  we  must  get 
feedback  from  a  cross-section  of  experience  levels  and  backgrounds. 

1 .  Age: _  (in  years) 

2.  Years  in  CF: _ 

3.  Years  as  a  pilot: _ 

4.  Handedness:  _ left  OR _ right 


Estimated  Flight  Time 


Type  of  Flight 

Estimated  time  (in  hours) 

1.  Total  flight  time 

2.  Total  rotary  wing 

3.  Total  Griffon 

4.  Total  time  flying  with  ERSTA 

5.  Time  spent  using  a  flight  simulator 

6.  How  familiar  are  you  with  the 

Gagetown  area  (very. . .  .not  at  all)? 

Specify  the  type  of  simulator  experiences  that  you  have  had: 


Indicate  any  other  flight  experience  that  you  believe  may  be  relevant  to  your  perfonnance  in  this 
experiment. 
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ANNEX  E 

CSE  Ratings:  Hand  vs.  Moving  Map  for  Flying  Pilot 
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TAMSS  Experiment  2 


Participant  ID: 


DAY  1:  Hand  Map  Alone  versus  Moving  Map  comparison 
FLYING  PILOT  VERSION 

Rate  the  impact  of  having  a  hand  map  alone  as  compared  to  a  moving  map  on  your  experiences 
in  the  simulator.  Circle  your  response  for  each  item.  A  rating  of  ‘4’  means  there  was  no 
difference  between  the  hand  map  alone  and  moving  map  conditions. 


COMPARISON 

Hand  Map  Moving 

Alone  much  Map  much 

better  than  No  difference  better  than 

Moving  Hand  Map 

Map  Alone 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

2.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

3.  Awareness  of  spatial 

orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

4.  Awareness  of  activities 

(enemies,  friendly  units) 

1 

2 

3 

4 

5 

6 

7 

NA 

5.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

NA 

6.  Using  information  from  the 

scene  to  control  the  aircraft 

1 

2 

3 

4 

5 

6 

7 

NA 

7.  Awareness  of  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

8.  Communication  from  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

NA 

9.  Communication  to  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

NA 

10.  Low-level  flight 

1 

2 

3 

4 

5 

6 

7 

NA 
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1 1 .  Low-level  maneuvering 

1 

2 

3 

4 

5 

6 

7 

NA 

12.  Maintaining  heading 

1 

2 

3 

4 

5 

6 

7 

NA 
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DAY  2:  Hand  Map  Alone  versus  Moving  Map+Sensor  comparison 
FLYING  PILOT  VERSION 


Rate  the  impact  of  having  a  hand  map  alone  as  compared  to  a  moving  map  and  the  sensor  on 
your  experiences  in  the  simulator.  Circle  your  response  for  each  item.  A  rating  of  ‘4’  means 
there  was  no  difference  between  the  hand  map  alone  and  the  moving  map  +  sensor  conditions 


COMPARISON 

Hand  Map 
Alone  much 
better  than 
Moving 
Map+Senso 
r 

No  difference 

Moving 
Map+Senso 
r  much  better 
than  Hand 
Map  Alone 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

2.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

3.  Awareness  of  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

4.  Awareness  of  activities  (enemies,  friendly 

1 

2 

3 

4 

5 

6 

7 

NA 

units) 

5.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

NA 

6.  Using  information  from  the  scene  to  control 

1 

2 

3 

4 

5 

6 

7 

NA 

the  aircraft 

7.  Awareness  of  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

8.  Communication  from  Mission  Commander 

1 

2 

3 

4 

5 

6 

7 

NA 

9.  Communication  to  Mission  Commander 

1 

2 

3 

4 

5 

6 

7 

NA 

10.  Low-level  flight 

1 

2 

3 

4 

5 

6 

7 

NA 

1 1 .  Low-level  maneuvering 

1 

2 

3 

4 

5 

6 

7 

NA 

12.  Maintaining  heading 

1 

2 

3 

4 

5 

6 

7 

NA 
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13.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

NA 

14.  Completing  the  mission 

1 

2 

3 

4 

5 

6 

7 

NA 
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ANNEX  F 

CSE  Ratings:  Hand  vs.  Moving  Map  for  Mission  Commander 
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DAY  1:  Hand  Map  Alone  versus  Moving  Map  comparison 
MISSION  COMMANDER  VERSION 


Rate  the  impact  of  having  a  hand  map  alone  as  compared  to  a  moving  map  on  your  experiences 
in  the  simulator.  Circle  your  response  for  each  item.  A  rating  of  ‘4’  means  there  was  no 
difference  between  the  hand  map  alone  and  moving  map  conditions. 


COMPARISON 

Hand  Map 
Alone  much 
better  than 

Moving 

Map  Alone 

No  difference 

Moving 
Map  much 
better  than 

Hand  Map 
Alone 

15.  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

16.  Keeping  track  of  activity 

1 

2 

3 

4 

5 

6 

7 

NA 

(enemies,  friendly  units) 

17.  Overall  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

18.  Using  the  map  to  navigate 

1 

2 

3 

4 

5 

6 

7 

NA 

19.  Planning  the  route 

1 

2 

3 

4 

5 

6 

7 

NA 

20.  Reading  the  map 

1 

2 

3 

4 

5 

6 

7 

NA 

21.  Using  the  CDU 

1 

2 

3 

4 

5 

6 

7 

NA 

22.  Using  comms 

1 

2 

3 

4 

5 

6 

7 

NA 

23.  Communication  to  flying 

1 

2 

3 

4 

5 

6 

7 

NA 

pilot 

24.  Communication  from  flying 

1 

2 

3 

4 

5 

6 

7 

NA 

pilot 

25.  Communication  to  ERSTA 

1 

2 

3 

4 

5 

6 

7 

NA 

operator 

26.  Communication  from  ERSTA 

1 

2 

3 

4 

5 

6 

7 

NA 

operator 

27.  Eyes  out  time 

1 

2 

3 

4 

5 

6 

7 

NA 

28.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 
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29.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

NA 

30.  Completing  the  mission 

1 

2 

3 

4 

5 

6 

7 

NA 
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TAMSS  Experiment  2  Participant  ID: 


DAY  2:  Hand  Map  Alone  versus  Moving  Map+Sensor  comparison 
MISSION  COMMANDER  VERSION 


Rate  the  impact  of  having  a  hand  map  alone  compared  to  a  moving  map  and  sensor  on  your 
experiences  in  the  simulator.  Circle  your  response  for  each  item.  A  rating  of  ‘4’  means  there  was 
no  difference  between  the  hand  map  alone  and  moving  map  +  sensor  conditions. 


COMPARISON 

Hand  Map 
Alone  much 
better  than 
Moving 

Map 

+Sensor 

No  difference 

Moving 
Map+Senso 
r  much  better 
than  Hand 
Map  Alone 

1)  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

2)  Keeping  track  of  activity 

1 

2 

3 

4 

5 

6 

7 

NA 

(enemies,  friendly  units) 

3)  Overall  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

4)  Using  the  map  to  navigate 

1 

2 

3 

4 

5 

6 

7 

NA 

5)  Planning  the  route 

1 

2 

3 

4 

5 

6 

7 

NA 

6)  Reading  the  map 

1 

2 

3 

4 

5 

6 

7 

NA 

7)  Using  the  CDU 

1 

2 

3 

4 

5 

6 

7 

NA 

8)  Using  the  comms 

1 

2 

3 

4 

5 

6 

7 

NA 

9)  Communication  to  flying 

1 

2 

3 

4 

5 

6 

7 

NA 

pilot 

10)  Communication  from  flying 

1 

2 

3 

4 

5 

6 

7 

NA 

pilot 

11)  Communication  to  ERSTA 

1 

2 

3 

4 

5 

6 

7 

NA 

operator 

12)  Communication  from 

1 

2 

3 

4 

5 

6 

7 

NA 

ERSTA  operator 
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13)  Eyes-outtime 

1 

2 

3 

4 

5 

6 

7 

NA 

14)  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

NA 

15)  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

NA 

16)  Completing  the  mission 

1 

2 

3 

4 

5 

6 

7 

NA 
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ANNEX  G 

Summary  of  Final  Questionnaire 
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Summary  of  Final  Questionnaire 
TAMSS  Project  -  Experiment  2 


Flying  Pilot  Role 


Paper  map  alone  versus  moving  map 

Overall  situation  awareness  5.33 

Overall  mission  awareness  5.33 

Awareness  of  spatial  orientation  5.67 

Awareness  of  activities  enemies,  friendly  units)  4.33 

Anticipating  future  events  3.67 

Using  information  from  the  scene  to  control  the  aircraft  5.33 
Awareness  of  heading  6 

Communication  from  Mission  Commander  5.33 

Communication  to  Mission  Commander  5.33 

Low-level  flight  5.67 

Low-level  maneuvering  5 

Maintaining  heading  5.67 


1  =  hand  map  alone  much  better,  4  =  no  difference,  7  =  moving  map  much  better 


Paper  map  alone  versus  moving  map  and  sensor 

Overall  situation  awareness  5 

Overall  mission  awareness  5 

Awareness  of  spatial  orientation  5 

Awareness  of  activities  (enemies,  friendly  units)  5 

Anticipating  future  events  4.33 

Using  information  from  the  scene  to  control  the  aircraft  5.33 
Awareness  of  heading  4 

Communication  from  Mission  Commander  5.33 

Communication  to  Mission  Commander  4.67 

Low-level  flight  5 

Low-level  maneuvering  4.67 

Maintaining  heading  4.33 


1  =  hand  map  alone  much  better,  4  =  no  difference,  7  =  moving  map  and  sensor  much  better 
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Mission  Commander  Role 


Paper  map  alone  versus  moving  map 


Overall  situation  awareness  5.33 

Keeping  track  of  activity  3.67 

(enemies,  friendly  units) 

Overall  spatial  orientation  5.67 

Using  the  map  to  navigate  6.67 

Planning  the  route  4.33 

Reading  the  map  5 

Using  the  CDU  3.33 

Using  coimns  4.67 

Communication  to  flying  pilot  5 

Communication  from  flying  pilot  4 

Communication  to  ERSTA  officer  NA 

Communication  from  ERSTA  officer  NA 

Eyes  out  time  5.33 

Overall  mission  awareness  5.33 

Anticipating  future  events  4.67 

Completing  the  mission  5.33 


1  =  hand  map  alone  much  better,  4  =  no  difference,  7  =  moving  map  much  better 


Paper  map  alone  versus  moving  map  and  sensor 


Overall  situation  awareness  6.33 

Keeping  track  of  activity  5.67 

(enemies,  friendly  units) 

Overall  spatial  orientation  6 

Using  the  map  to  navigate  5.67 

Planning  the  route  4 

Reading  the  map  5.33 

Using  the  CDU  4.33 

Using  coimns  4.33 

Communication  to  flying  pilot  5.67 

Communication  from  flying  pilot  4.33 

Communication  to  ERSTA  officer  5.67 

Communication  from  ERSTA  officer  5.67 

Eyes  out  time  4 

Overall  mission  awareness  6.33 

Anticipating  future  events  5.33 

Completing  the  mission  6.33 


1  =  hand  map  alone  much  better,  4  =  no  difference,  7  =  moving  map  and  sensor  much  better 
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Centre  for  Applied  Cognitive  Research 
March  31,  2004 

EXECUTIVE  SUMMARY 

As  part  of  the  Department  of  National  Defence  (DND)  Tactical  Aviation  Mission  System 
Simulation  (TAMSS)  Situation  Awareness  initiative,  the  Centre  for  Applied  Cognitive  Research 
(CACR)  at  Carleton  University  has  proposed  a  Cognitive  Systems  Engineering  (CSE) 
framework  to  be  used  as  a  guide  for  conducting  and  interpreting  evaluations  in  Modelling  and 
Simulation  (M&S)  programs.  The  present  document  is  a  report  on  Experiment  3  (E3)  of  the 
TAMSS  SA  program. 

Experiment  3  of  the  TAMSS  SA  program  had  three  major  activities.  The  first  activity  was  to 
further  integrate  and  extend  the  DND  CH-146  ERSTA-like  model  and  control  capabilities  with 
the  simulator  facility  at  the  CACR.  The  second  major  activity  was  to  provide  a  more  stable 
HLA-based  distributed  simulation  environment,  including  refinements  to  the  data  collection 
capabilities  of  the  CACR  simulator  facility.  The  third  major  activity  was  to  conduct  a  study  to 
further  test  the  CSE  framework  by  examining  the  perfonnance,  situation  awareness  (SA)  and 
workload  of  the  CH-146  Mission  Commander  (MC)  under  conditions  where  a  Mission  Specialist 
(MS)  was  present  versus  a  conditions  were  a  Mission  Specialist  was  not  present. 

Experiment  3  clearly  demonstrates  the  advantage  of  using  the  CSE  framework  for  M&S 
programs.  The  CSE  framework  provided  a  comprehensive  set  of  measures  that  allowed  a  broad 
overview  of  how  the  MS  affected  the  task.  Furthermore,  the  use  of  both  subjective  and  objective 
measures  of  SA,  workload,  and  performance  allowed  confirmation  across  the  various  aspects  of 
the  situation.  Specifically,  the  results  provide  a  clear  picture  of  how  the  presence  of  a  mission 
specialist  affected  the  subjective  and  objective  task  perfonnance.  In  particular,  the  MC,  freed 
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from  the  increased  demands  of  operating  the  sensor,  had  more  mental  attention  to  put  towards 
the  primary  demands  of  the  MC  role. 

The  experiment  was  also  technically  progressive.  The  DND  ERSTA-like  model  was 
successfully  extended  and  provided  the  requisite  level  of  functionality  to  enable  the  aircrew  to 
use  the  digital  moving  map  and  the  sensor  capabilities  in  a  realistic  and  appropriate  manner.  The 
distributed  simulation  environment  was  robust  and  stable:  The  High-Level  Architecture  (HLA) 
simulation  which  connected  the  ERSTA  system  model  to  the  flight  simulator  ran  flawlessly  for 
eight-ten  hours  per  day  across  twelve  days.  Furthermore,  the  complete  suite  of  data  collection 
utilities  was  stable  and  accurate  throughout  the  experiment. 

It  is  important  to  note  that  the  ERSTA-like  model  that  was  used  in  this  experiment  was  intended 
to  represent  the  primary  functionality  of  the  ERTSA  system  that  has  been  specified  as  a  possible 
technology  for  the  CH-146  Griffon.  However,  the  ERTSA-like  model  that  was  developed  was 
not  intended  as  a  prototype  ERSTA  system.  The  ERSTA-like  model  was  not  intended  to  provide 
the  full  functionality  and  capability  that  has  been  specified  for  the  CH-146  ERTSA  system.  For 
example,  the  magnification  (zoom)  in  the  ERSTA-like  model  was  significantly  less  than  that 
specified  for  the  ERSTA  system.  In  addition,  thermal  imaging  was  not  modeled:  only  a  camera 
sensor  was  provided.  Finally,  while  the  sensor  controls  were  reasonably  close  to  what  might  be 
expected  for  the  MS  station,  the  sensor  controls  that  were  provided  to  the  MC  were  not  intended 
to  match  the  controls  that  would  be  acquired  in  the  CH-146  ERSTA  program. 

It  is  concluded  that  the  CSE  framework  and  the  simulation  environment  that  was  developed  in 
the  TAMSS  SA  project  can  be  used  to  affect  the  design,  prototype,  test,  build  and 
implementation  processes  in  simulation-based  acquisition  programs. 
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Using  a  Cognitive  Systems  Framework  as  a  Guide  for  Modelling  and 

Simulation  Programs 

The  Impact  of  a  Mission  Specialist  on  the  Situation  Awareness,  Workload  and 
Performance  of  the  CH-146  Mission  Commander 


1  Background 

This  document  is  a  report  on  Experiment  3  (E3)  of  the  TAMSS  SA  program.  The  primary  goal 
of  this  experiment  was  to  exercise  and  evaluate  a  Cognitive  System  Engineering  (CSE) 
framework  for  assessing  the  impact  of  novel  technology  on  aircrew  in  the  CH-146  Griffon 
helicopter. 

The  current  experiment  follows  from  Experiment  2  (E2)  of  the  Tactical  Aviation  Mission  System 
Simulation  (TAMSS)  Situation  Awareness  (SA)  project.  In  E2,  CH-146  Griffon  aircrew 
consisting  of  a  Flying  Pilot  (FP)  and  a  Mission  Commander  (MC)  completed  a  series  of  zone 
reconnaissance  (recce)  missions.  Of  primary  interest  was  how  an  ERSTA-like  system,  including 
a  digital  moving  map  and  a  sensor  capability,  affected  the  performance,  situation  awareness  and 
workload  of  the  MC  during  the  execution  of  the  missions. 

Three  conditions  were  compared  in  E2: 

•  Paper  Map  (P-Map). 

This  was  a  baseline  condition  to  reflect  the  current  situation  in  the  Griffon  where 
aircrew  (i.e.,  the  MC)  navigate  using  a  hand-held  paper  map  and  detect  and 
identify  targets  without  aid  of  a  sensor. 

•  Moving  Map  (M-Map) 

In  this  condition,  the  MC  was  provided  with  a  digital  moving  map  positioned  on 
the  MC’s  leg/lap.  A  paper  map  was  also  provided  for  use  at  the  discretion  of  the 
MC.  As  in  the  paper  map  condition,  the  aircrew  perfonned  the  missions  without 
the  aid  of  the  ERSTA-like  sensor. 
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•  Moving  Map  plus  Sensor  (M-Map/Sensor) 

In  this  condition,  both  the  digital  moving  map  (and  the  paper  map)  and  the 
ERSTA-like  sensor  capability  were  provided  to  the  MC.  The  sensor  (camera) 
image  was  displayed  on  the  front  centre  console  in  the  position  where  the  Griffon 
FLIR  image  is  normally  located.  In  this  condition,  the  MC  was  able  to  use  the 
sensor  image  to  support  target  detection  and  identification. 

The  results  from  E2  showed  that  the  ERSTA-like  system  (digital  moving  map  and  sensor)  added 
significant  capability  to  Griffon  aircrew.  Participant  ratings  indicated  that  the  digital  moving 
map  and  sensor  enhanced  the  MC’s  perfonnance  and  SA  while  lowering  task  difficulty  and 
workload.  There  was  some  indication  of  these  benefits  being  transferred  to  the  FP,  particularly 
in  terms  of  the  aircrew’s  ability  to  position  the  aircraft  and  to  maintain  tactical  flight.  In 
addition,  head  tracking  data  showed  that  relative  to  the  P-Map  condition,  the  MC  was  able  to 
spend  more  time  looking  up  and  out  of  the  cockpit  in  the  M-Map  and  M-Map/Sensor  conditions. 

Experiment  2  of  the  TAMSS  SA  project  provided  a  solid  foundation  for  developing  the  current 
experiment.  Of  importance  in  E2  was  that  (a)  a  base-level  ERSTA-like  system  was  effectively 
modeled  and  integrated  into  the  simulator  environment  using  High-Level  Architecture  (HLA) 
protocol,  (b)  the  scenarios  that  were  developed  represented  realistic  and  workable  tactical 
missions,  and  (c)  the  questionnaire  battery  developed  for  obtaining  subjective  measures  proved 
to  be  sufficiently  sensitive  such  that  differences  in  performance,  SA,  and  workload  could  be 
detected  across  the  three  experimental  conditions. 

2  Introduction  to  Present  Experiment 

The  primary  goal  of  this  experiment  was  to  exercise  and  evaluate  a  Cognitive  System 
Engineering  (CSE)  framework  for  assessing  the  impact  of  novel  technology  on  aircrew  in  the 
CH-146  Griffon  helicopter.  Trained  CH-146  aircrew  completed  a  series  of  recce  missions.  On 
half  of  the  missions  the  crew  included  a  Flying  Pilot  (FP),  Mission  Commander  (MC),  and  a 
Mission  Specialist  (MS).  On  the  other  half  of  the  missions,  the  MS  was  not  included.  Of 
interest  was  how  the  presence  versus  absence  of  the  MS  affected  the  performance,  situation 
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awareness,  and  workload  of  the  CH-146  MC  while  completing  the  recce  missions. 

2.1  Major  activities 

The  conduct  of  Experiment  3  consisted  of  three  major  activities.  The  first  two  major  activities 
involved  simulation  engineering  to  enhance  the  ERSTA-like  system  that  was  modeled  and  used 
in  the  experiment  and  to  significantly  improve  the  stability  and  data  collection  facility  of  the 
simulation  environment.  The  third  major  activity  was  to  design  and  conduct  the  experiment. 

2.1.1  Extend  ERSTA-like  model 

The  first  major  activity  was  to  extend  the  functionality  and  control  capabilities  of  the  ERSTA- 
like  system  beyond  those  that  were  initially  modeled  in  E2  and  to  enhance  the  integration  of  this 
model  with  the  simulation  environment.  A  summary  of  the  ERSTA-like  system,  hardware  and 
software  architecture  is  presented  in  the  TAMSS  SA  final  report. 

Extending  the  ERSTA-like  system  included  modifying  the  model  that  was  used  in  Experiment  2 
(E2)  to  provide: 

•  additional  functionality  to  the  digital  moving  map  display,  including  a  military  grid 
overlay  for  the  digital  map,  touch  accessible  grid  read-out  capabilities  from  the  digital 
moving  map,  and  user  options  for  using  North-up  versus  heading-up  orientation,  and 

•  control  capability  of  the  sensor  image  for  the  Mission  Commander  (MC). 


2.1.2  Enhance  stability  of  simulation  environment. 

The  second  major  activity  in  E3  was  to  improve  the  stability  and  utility  of  the  simulation 
environment.  The  hardware  and  simulation  architecture  of  the  simulation  environment  as  well  as 
the  development  and  issues  surrounding  data  collection  in  a  distributed  system  are  presented  in 
the  TAMSS  SA  final  report. 
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The  enhancements  to  the  simulation  environment  included: 

•  improving  the  fundamental  stability  and  performance  of  the  ERSTA-like  model, 

•  stabilizing  the  flight  simulator  through  programming  upgrades  and  modification  of 
the  core  simulation  and  HLA  software, 

•  further  integration  of  the  ERTSA-like  model  and  the  flight  simulator,  and 

•  improvements  to  the  data  collection  capabilities  within  the  distributed  simulation 
environment. 


2.1.3  Design,  conduct  and  analysis  of  experiment 

The  third  major  activity  was  to  design,  conduct  and  analyze  the  experiment.  The  primary  goal  of 
this  experiment  was  to  detennine  whether  the  CSE  framework  could  be  used  to  measure  the 
impact  of  the  ERSTA-like  system  on  the  CH-146  aircrew,  and  in  particular,  on  the  perfonnance, 
SA,  and  workload  of  the  CH-146  MC.  To  this  end,  the  following  two  conditions  were  compared 
in  this  experiment: 

•  Mission  Specialist  Present.  In  this  condition,  the  crew  included  a  Mission 
Commander  (MC),  Flying  Pilot  (FP),  and  a  Mission  Specialist  (MS).  The  MS 
assumed  primary  operation  of  the  ERSTA-like  system,  in  and  particular,  the  sensor. 
The  MC  was  able  to  view  and  interact  with  the  digital  moving  map  and  if  desired, 
take  control  of  the  sensor. 

•  No  Mission  Specialist  Present.  In  this  condition,  the  crew  consisted  of  the  MC  and 
FP.  A  MS  was  not  present.  In  this  condition,  the  MC  assumed  responsibility  for 
operating  the  ERSTA-like  system. 
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The  conduct  of  E3  was  enabled  by  the  major  engineering  activities,  as  describe  above.  In 
addition,  E3  was  supported  by  the  following  activities: 

•  input  from  Subject  Matter  Experts  (SME)  regarding  the  functionality  and  use  of  the 
ERSTA-like  system  as  well  as  how  mission  specialists  could  be  integrated  into  the 
CH-146  aircrew, 

•  modification  of  the  tactical  scenarios  that  were  used  in  E2  of  the  TAMSS  SA  project 
in  order  to  provide  Fire  Mission  Support  (FMS)  capabilities  in  the  scenarios, 

•  development  of  tactical  knowledge  and  the  expertise  to  allow  for  dynamic  control  of 
elements  by  the  experimenters  during  the  missions,  including  the  escalation  of  enemy 
activity,  and 

•  modifications  of  the  questionnaire  battery  from  E2  that  were  used  for  obtaining 
subjective  ratings  of  performance,  situation  awareness,  and  workload. 

3  Method 

3.1  Participants:  Pilots 

The  background  questionnaire  for  the  pilots  is  shown  in  Annex  C.  Participants  were  7  male 
pilots  and  1  female  pilot  from  the  Canadian  Forces  ranging  in  age  from  30  to  48  years  (M  = 
40.125).  Years  in  the  Canadian  Forces  ranged  between  1 1  and  29  years  (M  =  20).  Years  as  a 
pilot  ranged  between  5  and  26  years  (M  =18.5).  Six  of  the  eight  pilots  were  right  handed  and 
two  pilots  were  left-handed.  All  eight  participants  had  some  experience  with  flight  simulators. 
These  included  full  motion  CH-146,  CH-135,  AH-64,  and  VH-1  simulators,  CT-1 14  Tutor 
Procedures  Trainer,  AH-64,  Bell  205/212/412,  Griffon  Beech  1900D,  Dash  8,  Airbus  320,  and 
Boeing  737.  The  following  table  summarizes  the  pilots’  estimated  time  in  hours  for  various 
types  of  flights. 
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Estimated  Time  (hours) 

Type  of  Flight 

Range 

Mean 

Total  flight  time 

1500  -9000 

4375 

Total  rotary  wing 

1200  -  7000 

3456 

Total  Griffon 

6  -  2500 

1182 

Time  spent  using  a  flight  simulator 

50  -  300 

156 

3.2  Participants:  Mission  Specialists 


The  background  questionnaire  for  the  mission  specialists  is  shown  in  Annex  D).  Mission 
specialists  were  2  males  from  the  Canadian  Forces  aged  44  and  46  years.  They  had  spent  25  and 
28  years  in  the  Canadian  Forces,  with  1  and  4  years  of  experience  as  mission  specialists.  Both 
were  right  handed.  Both  of  these  participants  had  some  experience  with  flight  simulators.  These 
included  7A/7B  simulators.  The  following  table  summarizes  the  mission  specialists’  estimated 
time  in  hours  for  various  times  of  flights. 


Estimated  Time  (hours) 

Type  of  Flight 

Range 

Mean 

Total  rotary  wing 

45  -  1700 

873 

Total  Griffon 

45-55 

50 

Time  spent  using  a  flight  simulator 

70 

70 

3.3  Measures 

In  Experiments  1  and  2  of  the  TAMSS  SA  project,  it  was  demonstrated  that  pilot’s  self  ratings  of 
their  performance,  SA  and  workload  can  provide  a  reasonable  index  concerning  the  impact  of  a 
new  cockpit  technology  (i.e.,  HUD).  Moreover,  in  El  it  was  demonstrated  that  SA  could  be 
objectively  assessed  by  measuring  a  pilot’s  ability  to  detect  and  report  airborne  (e.g.,  other 
aircraft)  and  ground  entities  (e.g.,  tanks,  downed  aircraft)  while  performing  a  mission.  An 
important  finding  from  El  was  that  this  objective  measure  revealed  significant  and  differences  in 
SA  even  in  conditions  where  the  pilots’  subjective  ratings  of  SA  were  not  different. 
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In  the  present  experiment,  all  three  dimensions  of  behaviour  outlined  in  the  CSE  framework 
were  measured  subjectively:  performance,  workload,  and  situation  awareness.  (See  Table  3.3 
below).  In  addition,  a  focus  was  placed  on  obtaining  an  objective  measure  of  workload  as  well 
as  objective  measures  of  perfonnance/behaviour. 

Table  3.3  CSE  Measures 


Type  ol 

'  Measure 

CSE  Domain 

Objective 

Subjective 

Task-Relevant 

Performance/Behaviour 

Head  positioning  of  MC 

Use  of  sensor 

Ratings  of  performance 

Situation  Awareness 

Detection  of  airborne  and 

ground  objects 

Ratings  of  SA  (specific  and 

global) 

Workload 

Detection  of  visual  stimuli 

Ratings  of  workload 

3.3.1  Task-relevant  performance  measure 

•  Subjective  ratings.  Following  each  mission,  the  FP  and  the  MC  completed  subjective 
ratings  of  performance  in  the  mission.  These  rating  questionnaires  are  shown  in 
Annex  B. 

•  Objective  measure  1:  Head-down  time.  One  objective  measure  was  the  head¬ 
positioning  of  the  MC.  It  was  hypothesized  that  when  a  MS  was  included  in  the 
crew,  the  MS  would  be  given  primary  responsibility  for  operating  the  ERSTA-like 
sensor.  For  missions  where  the  crew  did  not  include  a  MS,  the  MC  was  required  to 
control  the  sensor  image.  It  was  predicted,  therefore,  that  MCs  would  spend  less  time 
with  their  heads  down  and  inside  the  cockpit  when  a  MS  was  present  as  compared  to 
when  a  MS  was  not  present. 
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•  Objective  measure  2:  Sensor  time.  A  second  objective  measure  of  task-relevant 
performance  was  the  amount  of  time  the  sensor  was  used  throughout  a  mission.  It 
was  predicted  that  the  crew’s  use  of  the  sensor  would  be  greater  when  a  MS  was 
present.  When  a  MS  was  not  present,  the  MC  would  have  limited  time  available  for 
controlling  the  sensor. 

3.3.2  Situation  awareness  measures 

•  Subjective  ratings.  Situation  awareness  was  measured  subjectively  in  this  experiment 
by  having  the  FP  and  MC  complete  Likert-scale  ratings  of  SA  after  each  mission  (see 
Annex  B) 

•  Objective  measures.  Objective  measures  of  SA  were  not  obtained  (but  see  El  of  the 
TAMSS  project). 

3.3.3  Workload  measures 

•  Subjective  ratings.  Following  each  mission,  the  FP  and  MC  completed  separate 
Likert-scale  questionnaires  of  workload  as  well  as  workload  ratings  based  on  a 
modified  NASA  TLX.  Subjective  ratings  for  global  workload  were  obtained  as  were 
ratings  for  specific  segments  (e.g.,  ingress,  recce-zone,  egress)  in  the  missions  (see 
Annex  B). 

•  Objective  measure.  Workload  was  objectively  assessed  using  a  visual  detection  task 
whereby  the  MC  was  required  to  indicate  when  they  detected  a  visual  target  (a  briefly 
displayed  green  circle)  on  the  front  screen.  The  targets  subtended  approximately  2 
deg  of  visual  angle  and  were  presented  every  15  sec  (+/-  3  sec  randomly  determined) 
throughout  the  workload  missions. 

3.4  Materials 

The  crew  completed  realistic  zone-recce  missions  (see  descriptions  in  Annex  A).  For  each 
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mission,  a  primary  objective  was  defined  and  details  were  given  concerning  the  flight  path 
leading  to  the  Release  Point  (RP).  Pre-mission  information  was  given  concerning  known 
position  and  movement  of  friendly  and  enemy  forces  in  the  area.  Aircrew  planned  their  post  RP 
routes  and  observation  points  in  pre-mission  sessions  lasting  between  30  -  60  minutes. 

Questionnaires.  The  subjective  measurements  of  situation  awareness,  workload,  and 
performance  were  administered  at  the  end  of  each  mission.  The  comparison  questionnaire  was 
administered  at  the  end  of  the  experiment.  For  each  mission,  separate  questionnaires  were 
completed  by  the  FP  and  the  MC.  Participants  rated  variables  on  a  number  of  scales  (refer  to 
Annex  B).  In  addition,  before  starting  the  experiment,  participants  completed  a  background 
questionnaire  (see  Annex  C),  which  included  questions  about  the  number  of  tactical,  Griffon,  and 
simulator  flying  hours  they  had  logged. 

Head  Position  of  MC.  Head  position  data  was  collected  online  (30  hz)  to  determine  where  the 
MC  was  looking  throughout  each  mission.  From  these  data,  percent  head-up  versus  head-down 
time  was  calculated. 

Development  of  Mission  Scenarios.  An  important  activity  in  this  experiment  was  to  further 
develop  the  tactical  scenarios  that  were  used  in  E2  of  the  TAMSS  SA  project.  In  particular,  the 
scenarios  from  E2  were  extended  to  provide  a  heightened  level  of  activity  in  the  recce  zone. 
Activity  included  possible  fire  from  the  enemy  as  well  as  the  ability  for  the  Griffon  crew  to  call 
in  a  Fire  Support  from  the  friendly  assets.  The  scenarios  were  also  evolved  to  allow  for  changes 
in  elements  such  as  the  location  of  the  FARP.  The  scenarios  also  included  an  enhanced  level  of 
radio  communications. 

Terrain  database.  The  landscape  database  was  a  Virtual  Reality  model  of  a  40  km  (east  to  west) 
by  50  km  (north  to  south)  section  of  CFB  Gagetown,  NB,  divided  into  twenty  10  km  by  10  km 
squares,  or  “tiles”.  Due  to  computer  system  (memory)  constraints,  only  6  to  8  tiles  (i.e.,  the 
minimum  required  to  cover  the  terrain  relevant  to  a  given  mission)  were  displayed  during  any 
one  mission.  The  database  contained  a  number  of  fixed,  pre-determined  geographical  features 
(river,  hills,  forest)  and  man-made  elements  (barracks,  various  military  installations,  roads,  and 
the  flight  base).  Various  entities,  both  moving  and  stationary,  were  added  to  the  terrain  database 
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to  create  a  number  of  mission  scenarios  (see  below)  in  order  to  assess  pilots’  situation  awareness 
during  missions. 


The  objects  that  were  placed  in  the  scenarios  included: 

(a)  formation  of  five  Ml  13  armored  personnel  carriers  (mission  scenario  1), 

(b)  one  CH-146  Griffon  helicopter  flying  in  a  circle  pattern  (scenario  1), 

(c)  formation  of  BMP-3  light  infantry  vehicles  (scenario  1), 

(d)  formation  of  six  Ml  025  light  tactical  vehicles  (all  scenarios), 

(e)  formation  of  eight  M- 1 09  artillery  vehicles  (scenarios  1  and  2), 

(f)  platoon  of  6  infantry  tents  (all  scenarios), 

(g)  formations  of  M2A3  Bradley  Fighting  Vehicles  (scenarios  1,2  and  4), 

(h)  formation  of  six  M1A2  battle  tanks  (scenario  2), 

(i)  damaged  M939A2  5-ton  truck  (scenario  2), 

(j)  formation  of  four  BTR-80  armoured  personnel  carriers  (scenario  2), 

(k)  formation  of  six  Ml  10  artillery  vehicles  (scenario  3), 

(l)  crashed  Mi-28  attack  helicopter  (scenario  3), 

(m)  formation  of  six  M270  self-propelled  rocket  launchers  (scenario  3), 

(n)  formations  of  HUM  Avenger  armoured  vehicles  (scenarios  1,  3  and  4), 

(o)  formation  of  five  Ml  045  TOW  missile  carrier  vehicles, 

(p)  downed  Su-25  ground  attack  airplane,  and 

(q)  formation  of  four  ZSU-23-4  self-propelled  anti-aircraft  guns. 

The  objects  were  placed  so  as  to  be  on  or  near  the  trajectories  or  Restricted  Operation  Zones 
(ROZ’s).  Most  entities  or  fonnations  were  stationary,  but  each  scenario  included  at  least  two 
formations  of  moving  vehicles  (these  tended  to  be  vehicle  formations  that  participants  had  not 
been  briefed  on  at  the  outset  of  their  missions).  All  entities  were  scaled  to  their  nonnal  size 
relative  to  the  database. 


3.5  Mission  Scenarios 


Four  mission  scenarios  were  used,  with  terrain  features  and  vehicles  as  described  above. 
Descriptions  of  the  scenarios  are  included  in  Annex  A.  The  missions  were  roughly  equivalent  in 
complexity  and  number  of  entities,  and  all  followed  a  general  schema.  Participants  were  first 
briefed  on  the  general  context  of  the  mission,  an  operation  that  was  designed  to  resemble  United 
Nations  peace  support  operations  that  have  been  put  into  action  on  various  fronts  throughout  the 
Balkans  and  the  Middle  East  in  recent  years. 

Each  mission  consisted  of  a  starting  point,  and  intermediate  waypoint,  a  release  point  (RP),  and  a 
Restricted  Operations  Zone  (ROZ)  that  contained  the  objective(s)  for  the  mission.  The  start 
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point,  intermediate  waypoint,  RP  were  connected  by  safe  flight  corridors  within  which  the 
aircrews  were  to  keep  the  ownship.  The  ownship’s  trajectory  within  the  ROZ  was  left  to  the 
discretion  of  the  pilots.1 

Each  mission  began  with  the  ownship  airborne  at  approximately  400  feet  above  ground  level  at 
the  start  point,  and  oriented  towards  the  first  waypoint.  The  waypoints  were  given  to  the  crew  on 
a  paper  map  and  were  displayed  on  the  ERSTA  digital  moving  map.  Participants  were  instructed 
to  maintain  a  maximum  altitude  of  500  feet  while  transiting  from  the  start/end  point  and  the 
ROZ,  and  a  maximum  of  250  feet  in  the  ROZ  (the  ROZ  was  assumed  to  be  capped  by  active 
high-speed  airspace).  In  practice,  the  crew  flew  at  much  lower  altitudes,  especially  in  the  recce 
zone  where  tactical  flight  was  maintained. 

In  each  mission,  the  crew  was  given  the  task  of  locating  and  assessing  the  state  of  a  primary 
objective  (generally  a  bridge  or  some  other  strategic  landmark).  The  crew  was  informed  in 
advance  of  the  entities  they  were  expected  to  encounter,  but  each  mission  included  several 
entities  that  were  not  briefed.  Also,  each  mission  included  unscripted  weapons  activity  (directed 
either  at  the  ownship  or  at  entities  or  geographic  locations  in  the  terrain)  that  was  controlled  on- 
the-fly  by  an  experimenter  using  the  STAGE  software. 

3.6  Procedure 

Two  Mission  Specialists  (MS)  participated  in  this  experiment.  One  MS  participated  with  the 
first  two  pairs  of  Griffon  crew  and  the  second  with  the  last  two  pairs  of  Griffon  crew.  Each  MS 
arrived  two  days  prior  to  the  CH-146  aircrew  in  order  to  gain  experience  operating  the  ERSTA- 
like  system  and  to  develop  procedures  related  to  the  functionality  and  use  of  the  system. 

For  each  pair  of  pilots,  experimentation  took  place  across  two  days.  Upon  arrival  at  the  lab  on 
day  one,  the  pilots  were  provided  with  infonnation  about  the  general  purpose  of  the  experiment 
and  were  given  an  overview  of  the  two-day  schedule.  The  pilots  then  completed  an  informed 
consent  and  the  personal  (experience)  background  questionnaire. 

1  The  ROZ  was  the  area  within  which  the  actual  reconnaissance  mission  was  supposed  to  take  place.  The 
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Following  the  day  one  information  session,  the  pilots  were  introduced  to  the  functionality  of  the 
ERSTA-like  system  that  was  modeled  for  this  experiment.  This  introduction  was  done  in 
combination  with  the  experimenters  and  the  MS  (who  had  two  days  prior  experience  in  the  lab 
working  with  the  ERSTA-like  system).  A  practice  session  was  then  conducted  wherein  one 
participant  served  as  the  FP  and  the  other  participant  served  as  the  MC.  The  practice  session  was 
conducted  until  the  FP  was  comfortable  flying  tactical  with  the  simulator  environment  and  the 
MC  was  familiar  with  the  functionality  and  use  of  the  ERSTA  digital  moving  map  and  sensor 
system  (including  control  of  the  sensor  image).  The  same  practice  procedure  was  completed  at 
the  start  of  day  two,  except  that  the  CH-146  pilots  switched  roles. 

Each  aircrew  flew  a  total  of  six  missions,  two  missions  on  day  one  and  four  missions  on  day  two. 
On  day  one,  one  pilot  served  as  the  FP  and  the  other  as  the  MC  for  the  first  two  missions.  On  day 
two,  the  roles  were  reversed  for  the  first  two  missions.  The  final  two  missions  on  day  two  were 
the  workload  missions  which  were  run  with  the  previous  day  one  assignment  of  FP  and  MC 
roles.  These  last  two  missions  were  designed  to  obtain  an  objective  measure  of  the  MC’s 
workload. 

Each  mission  took  approximately  20-30  minutes  to  complete.  After  each  mission,  the 
participants  were  asked  to  fill  out  the  mission  questionnaires.  At  the  end  of  the  second  day  the 
participants  were  asked  to  fill  out  a  final  questionnaire. 

The  first  of  each  pair  of  missions  was  conducted  with  the  MS  present.  The  second  of  each  pair 
of  missions  was  conducted  without  a  MS. 

3.7  Research  Personnel 

Four  experimenters  assisted  with  running  the  experiment.  Experimenter  #1  provided  online 
coordination  of  the  scenarios  and  of  the  other  experimenters.  This  experimenter  monitored  the 
progress  of  the  scenario  and  decided  when  radio  contacts  should  be  directed  to  the  crew  from 
agents,  such  as  Brigade  Headquarters  (92)  or  a  Recce  Patrol  (T43B).  Experimenter  #1  also 

participants  were  therefore  given  free  reign  to  set  up  observation  points  within  the  ROZ. 
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decided  when  enemy  activity  would  be  initiated  and  when  and  specifically  where  the  enemy 
would  engage  in  aggressive  action  (e.g.,  fire  upon  the  objective  or  in  the  vicinity  of  the  ownship). 

Experimenter  #2  provided  radio  contact  with  the  crew  by  serving  at  the  voice  of  BDE 
Headquarters  (92),  the  Recce  Patrol  (T43B),  or  friendly  artillery  (G24),  on  a  need  basis  in 
response  to  contact  from  the  crew  and  when  directed  by  Experimenter  #1.  Experimenter  #3 
controlled  specific  events  in  the  scenarios  using  STAGE.  Experimenter  #4  directed  the  input  of 
specific  data  collection  information.  This  experimenter  was  also  responsible  for  the  calibration 
and  monitoring  of  the  head-tracking  system  and  the  general  status  of  the  simulation. 

4  Results 

The  experimental  goals  were  to  extend  the  evaluation  of  the  Cognitive  System  Engineering 
(CSE)  framework  proposed  by  the  CACR  for  evaluating  the  impact  of  novel  technology  on 
aircrew  in  the  CH-146  Griffon  helicopter.  To  do  this,  the  impact  of  an  ERSTA-like  system  was 
assessed  with  a  particular  focus  on  how  ERSTA  affects  the  performance,  situation  awareness  and 
workload  of  the  MC.  Two  conditions  were  examined:  Mission  Specialist  Present  versus  No 
Mission  Specialist. 

The  data  from  the  post-mission  questionnaires  is  listed  in  Annex  E. 

4.1  Task-Relevant  Performance 

4.1.1  Flying  pilot  role 

Subjective  performance  ratings.  Subjective  ratings  of  performance  for  the  FP  role  are  shown  in 
Figure  4. 1 .  Lower  values  reflect  worse-rated  performance  where  higher  values  reflect  better¬ 
rated  performance.  Overall,  FP  perfonnance  was  self-rated  as  “good”. 

In  the  FP  role,  participants  rated  their  control  of  airspeed  and,  more  interestingly,  their 
positioning  the  aircraft  in  the  recce  zone,  as  being  significantly  better  when  a  MS  was  present  as 
compared  to  when  no  MS  was  present. 
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The  impact  of  the  MS  on  positioning  the  aircraft  reflects  two  factors.  First,  when  a  MS  was 
present  the  MC  had  more  mental  resources  to  allocate  toward  directing  the  FP  into  position. 
Second,  the  FP  was  able  to  monitor  information  from  the  MS  and  when  appropriate  position  the 
aircraft  accordingly.  A  typical  example  was  when  the  MS  indicated  that  the  sensor  could  not  be 
placed  on  a  target  or  objective  because  the  aircraft  was  too  low.  In  these  cases,  the  MC  would,  at 
their  discretion,  direct  the  FP  to  either  provide  more  altitude  or  to  briefly  pop  up.  During  these 
maneuvers  the  MS  would  indicate  when  the  sensor  had  sight,  thereby  giving  direct  feedback  that 
the  FP  could  use  to  modulate  their  actions. 


Figure  4,1 
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Flying  Pilot:  Performance  Ratings 
1  =  low  7  =  high 


□  No  MS 

□  MS  Present 


Subjective  difficulty  ratings.  Figure  4.2  shows  subjective  ratings  of  difficulty  in  perfonning  the 
same  tasks  as  those  assessed  in  the  above  perfonnance  ratings.  Lower  values  reflect  less 
difficulty  perfonning  tasks  whereas  higher  values  reflect  more  difficulty  performing  tasks. 
Overall,  difficulty  in  performing  FP  tasks  was  self-rated  as  slightly  less  than  of  moderate 
difficulty.  There  were  no  significant  differences  in  rated  difficulty  between  the  MS-present 
versus  the  No-MS  conditions. 
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Figure  4.2 


Flying  Pilot:  Task  Difficulty  Ratings 
1  =  easy  7  =  hard 


7 

6 

*7  5 
ui  4 

I  > 

2 

1 


Finding  Control  Control  Control  Cross 

waypoints  heading  altitude  airspeed  checking 


Positioning  Maintain 
Aircraft  tactical  flight 


□  No  MS 

□  MS  Present 


4.1.2  Mission  commander  role 


Subjective  performance  ratings.  Subjective  ratings  of  perfonnance  for  the  MC  role  are  shown  in 
Figure  4.3,  first  six  pairs  of  bars  starting  from  the  left.  Lower  values  reflect  worse  rated 
performance  where  higher  values  reflect  better  rated  performance. 


Self-rated  MC  performance  varied  considerably  from  “slightly  less  than  adequate”  to  “good”, 
depending  on  the  task.  For  the  MC  role,  participants  rated  their  perfonnance  on  all  but  two  tasks 
(using  comins  and  positioning  the  sensor)  as  being  better  when  a  MS  was  present  as  compared  to 
when  no  MS  was  present. 
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Figure  4.3 


Mission  Commander:  Performance  Ratings 


Subjective  difficulty  ratings.  Figure  4.4  shows  subjective  ratings  of  difficulty  in  perfonning  the 
same  tasks  as  those  assessed  in  the  above  perfonnance  ratings.  Lower  values  reflect  less 
difficulty  perfonning  tasks  whereas  higher  values  reflect  more  difficulty  performing  tasks. 

Overall,  difficulty  in  performing  the  MC  tasks  was  self-rated  from  “slightly  less  than  moderate 
difficulty”  to  “slightly  more  than  moderate  difficulty”.  Overall,  task  difficulty  was  rated  as 
higher  when  in  the  No-MS  condition  than  in  the  MS-present  condition.  As  shown  in  Figure  4.4 
large  (and  all  t-tests  statistically  significant  at p  <  .05)  self-rated  difference  in  difficulty  were 
found  for  “using  the  comins”,  “use  of  the  sensor  capability”,  “positioning  the  sensor”,  “guiding 
the  FP  to  position  the  aircraft”,  and  “getting  information  from  the  ERSTA  system”. 
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Figure  4.4 


Mission  Commander:  Task  Difficulty  Ratings 
1  =  easy  7  =  hard 
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□  NO  MS 

□  MS  Present 


Objective  measure  of performance/behaviour:  of  head-positioning.  Head  position  data  for  the 
MC  was  collected  throughout  each  mission.  As  shown  in  Figure  4.5,  overall  percent  head-up 
time  for  the  MC  was  significantly  greater  (better)  when  a  MS  was  present  as  compared  to  when  a 
MS  was  not  present  (37.1%  vs.  20.1%  ,  respectively,  t(6)  =  3.91,  p  <  .008).  Head-up  time  should 
impact  on  flight  safety  and  performance:  Enhanced  head-up  time  should  facilitate  the  MC’s 
ability  to  detect  and  respond  to  infonnation  external  to  the  cockpit.  A  closer  examination  of  the 
data  in  Figure  4.5  shows  that  when  there  was  no  MS  present,  the  MC  spent  more  time 
(approximately  6%)  looking  at  the  sensor  image  and  the  digital  moving  map.  This  extra  time  on 
the  sensor  was  likely  due  to  the  additional  requirement  on  the  MC  to  operate  the  sensor  when 
there  was  no  MS:  Operating  the  sensor  requires  frequent  use  of  the  digital  map  orienting  and 
moving  the  sensor  (touch-click  operation).  These  findings  concurs  with  the  subjective  ratings  of 
difficulty  where  participants  indicated  that  “use  of  the  sensor”,  “positioning  the  sensor”  and 
“getting  infonnation  from  the  sensor”  was  quite  difficult  in  the  No-MS  condition. 
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Figure  4.5 


Mission  Commander:  Head  Position 
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Objective  measure  of  performance/behaviour:  sensor  usage.  Figure  4.6  shows  the  average 
percent  of  time  that  the  sensor  was  used  (being  moved)  relative  to  the  overall  mission  time.  As 
shown  by  the  far-left  data  bar,  when  a  MS  was  present  the  sensor  was  moved  by  the  MS  for  an 
average  of  40.4%  of  the  overall  mission  time.  The  middle  data  bar  in  Figure  2  shows  that,  on 
average,  the  sensor  was  being  controlled  by  the  MC  only  1 .7%  of  the  time  when  a  MS  was 
present:  thus,  the  MS  had  the  primary  responsibility  for  moving  the  sensor. 

The  far-right  data  bar  in  Figure  4.6  shows  that  when  there  was  no  MS  present,  the  sensor  was 
used  by  the  MC  for  an  average  of  26.6%  of  the  overall  mission  time.  Thus,  when  there  was  no 
MS,  the  sensor  was  used  less  then  half  of  the  time  compared  to  conditions  where  a  MS  was 
included  as  part  of  the  aircrew.  All  three  of  the  statistical  pairwise  comparisons  of  the  sensor 
usage  shown  in  Figure  4.6  were  significant  (p  <  .05). 
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Figure  4.6 
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4.2  Situation  Awareness 
4.2.1  Flying  pilot  role 

Subjective  ratings  of  SA.  Subjective  ratings  of  SA  for  the  FP  role  are  shown  in  Figure  4.7  (SA 
for  aircraft  system),  Figure  4.8  (SA  for  tactical  information,  and  Figure  4.9  (SA  for  Spatial 
Orientation).  Figure  4.9  also  include  ratings  for  the  crew  activity.  Overall,  the  FPs  rated  their 
SA  as  “moderate  to  good”. 

In  the  FP  role,  only  one  rating  of  SA  for  aircraft  systems  differed  significantly,  where  the  SA 
ratings  were  higher  in  the  MS-present  than  the  No-MS  condition.  This  rated  SA  for  “heading” 
(t( 7)  =  2.65,  p  <  .034;  Figure  4.7). 

A  significant  difference  for  Spatial  SA  and  Crew  Activity  was  found  for  the  rated  “usefulness  of 
information  provided  by  the  ERSTA  operator  (t(7)  =  3.75, p  <  .008)  (see  Figure  4.9).  This 
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finding  shows  that  the  FPs  found  the  ERSTA-related  information  to  be  more  useful  when  the  MS 
was  operating  the  ERSTA  system  than  then  the  MC  was  operating  the  ERSTA  system.  Related 
to  this  is  a  near-significant  difference  in  rated  SA  for  tactical  awareness,  and  in  particular  the  FP 
rated  ability  to  anticipate  future  developments  (t(7)  =  2.18,  p  <  .065;  Figure  4.8). 


Figure  4.7 


Flying  Pilot:  Rated  SA  for  Aircraft  Systems 
1  =  lowSA  7  =  high  SA 


□  NO  MS 

□  MS  Present 


Figure  4.8 


Flying  Pilot:  Rated  SA  for  Tactical  Information 
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Figure  4.9 

Flying  Pilot:  Rated  Spatial  SA  &  Crew  Activity 
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4.2.2  Mission  commander  role 

Subjective  ratings  of  SA.  Subjective  ratings  of  SA  for  the  MC  role  are  shown  in  Figures  4.10 
(tactical  SA),  4.1 1  (spatial  SA)  and  4.12  (crew  activity  SA).  Overall,  MC  SA  was  self-rated  as 
“moderate  to  good”.  For  the  MC  role,  SA  was  generally  rated  as  being  significantly  higher  in 
the  MS-present  than  in  the  No-MS  condition.  This  is  true  for  the  MCs’  ratings  of  tactical 
awareness,  spatial  awareness,  and  crew  awareness. 

Figure  4.10 


Mission  Commander:  Rated  Tactical  SA 
1  =  low  SA  7  =  high  SA 
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□  MS  Present 
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Figure  4.11 


Mission  Commander:  Rated  Spatial  SA 
1  =  low  SA  7  =  high  SA 
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Figure  4.12 


Mission  Commander:  SA  for  Crew  Activity 
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4.3  Workload 

The  workloads  for  the  FP  and  the  MC  were  subjectively  measured  using  two  scales:  a  7-point 
Likert  scale  and  a  modified  NASA  TLX  scale.  An  objective  measure  of  the  MC’s  workload  was 
also  obtained. 

4.3.1  Flying  pilot  role 

Subjective  ratings  of  workload:  Likert  scale.  Subjective  Likert-scale  ratings  of  SA  for  the  FP  role 
are  shown  in  Figure  4.13.  Overall,  FP  workload  was  self-rated  as  “low  to  moderate”.  There 
were  no  significant  differences  in  self-rated  workload  between  the  MS-present  and  the  No-MS 
conditions. 


Figure  4.13 


Flying  Pilot:  Rated  Workload  (Likert  Scale) 
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Subjective  ratings  of  workload:  NASA  TLX.  Subjective  TLX  ratings  of  SA  for  the  FP  role  are 
shown  in  Figure  4.14.  The  five  quadrants  in  Figure  xx  represent  TLX  ratings  associated  with  the 
five  mission  segments:  ingress  to  RP,  RP  to  first  observation  point,  observing  a  target,  activity  in 
the  recce  zone,  and  egress. 

The  pattern  of  workload  for  the  FP  is  as  expected:  overall,  workload  was  rated  as  lowest  during 
ingress  and  egress  and  highest  during  activity  in  the  recce  zone.  There  was  only  one  comparison 
for  which  the  FPs’  workload  was  self-rated  to  be  higher  in  the  No-MS  than  the  MS-present 
condition:  temporal  demand  when  getting  and  maintaining  observation  (3’rd  quadrant  in  figure) 
of  a  target. 

The  TLX  assessment  also  included  one  question  per  mission  segment  concerning  the  rated  level 
of  performance.  These  are  the  spikes  in  the  graph  as  FP  performance  was  generally  rated  as 
quite  high.  In  addition,  FP  performance  was  self-rated  to  be  higher  in  the  MS-  Present  than  the 
No-MS  condition  for  getting  and  maintaining  observation  of  a  target  and  for  level  of  overall 
performance  activity  in  the  recce  zone. 
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Figure  4.14 


Flying  Pilot:  TLX  Workload  Ratings 

Hngress  RP=Release  Point  Ob  Observation  points 
Rc=RecceZone  E=Egress 
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4.3.2  Mission  commander  role 

Subjective  ratings  of  workload:  Likert  scale.  Subjective  Likert-scale  ratings  of  SA  for  the  MC 
role  are  shown  in  Figure  4.15.  Overall,  MC  FP  workload  was  self-rated  as  “slightly  less  than 
moderate”.  There  were  several  significant  differences  in  self-rated  workload  between  the  MS- 
present  and  the  No-MS  conditions.  Of  particular  note  where  higher  rated  workloads  in  the  No- 
MS  condition  that  were  associated  with  communicating  with  the  FP  (7(1 1)  =  3.63,/?  <  .004)  and 
with  the  MS  (f(l  1)  =  2.98,  p  <  .012),  guiding  the  MS  (t(l  1)  =  3.23,  p  <  .008),  positioning  the 
sensor  (t(10)  =  3.15,/?  <  .01),  radio  communications  (f(10)  =  2.67, p  <  .024),  and  overall 
workload  (t(l  1)  =  2.28, p  <  .044). 

Figure  4.15 


Mission  Commander:  Rated  Workload  (Likert  Scale) 
1  =  low  workload  7  =  high  workload 
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Subjective  ratings  of  workload:  NASA  TLX.  Subjective  TLX  ratings  of  SA  for  the  MC  role  are 
shown  in  Figure  4.16.  The  five  quadrants  in  Figure  4.16  represent  TLX  ratings  associated  with 
the  five  mission  segments:  ingress  to  RP,  RP  to  first  observation  point,  observing  a  target, 
activity  in  the  recce  zone,  and  egress.  The  pattern  of  workload  for  the  MC  is  as  expected: 
workload  was  rated  as  lowest  during  ingress  and  egress  and  highest  during  activity  associated 
with  observing  targets  and  activity  in  the  recce  zone. 

As  shown  in  Figure  4.16,  for  the  ingress  ,  RP  to  observation,  and  egress  segments,  there  were  no 
significant  differences  in  rated  TLX  workload  between  the  No-MS  versus  the  MS-Present 
conditions.  In  the  observation  segment,  self-ratings  were  higher  in  the  No-MS  than  the  MS- 
present  condition  for  frustration  (t(l  1)  =  4.25,/?  <  .001),  effort  (7(1 1)  =  2.43,/?  <  .034)  and 
performance  (t(l  1)  =  4.53,/?  <  .001).  In  the  recce  zone  segment,  self-ratings  were  higher  in  the 
No-MS  than  the  MS-present  condition  for  temporal  demand  (t(l  1)  =  2.65,/?  <  .023),  frustration 
(t(l  1)  =  2.47,/?  <  .031),  effort  (f(l  1)  =  2.05,/?  <  .06),  and  performance  (t(l  1)  =  3.47,/?  <  .005). 
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Figure  4.16 


Mission  Commander:  TLX  Workload  Ratings 

□  No  MS  □  MS  Present 


Objective  measure  of  workload.  The  MCs’  workload  was  objectively  assessed  using  a  visual 
detection  task  whereby  the  MC  was  required  to  indicate  when  they  detected  a  visual  target  (a 
briefly  displayed  green  circle)  on  the  front  screen.  The  targets  subtended  approximately  2  deg  of 
visual  angle  and  were  presented  every  15  sec  (+/-  3  sec  randomly  determined)  throughout  the 
workload  missions.  The  MC  was  required  to  push  a  foot  switch  whenever  they  detected  a  visual 
target.  The  measure  of  performance  was  percent  visual  targets  that  were  detected. 
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The  MC  responses  to  the  visual  targets  was  divided  into  two  classes  of  activity:  transit  versus 
observation.  The  transit  category  includes  the  MC  responses  to  the  visual  targets  when  the  crew 
was  engaged  in  the  initial  ingress,  transit  from  the  RP  to  the  first  observation  point,  moving  from 
one  observation  point  to  another  observation  point,  and  egress.  The  observation/contact  category 
refers  to  MC  responses  to  the  visual  targets  when  the  crew  was  observing  a  target/objective, 
submitting  a  contact  report,  or  perfonning  a  FSM.  As  shown  in  Figure  4.17,  MCs  detected  most 
(average  of  82%)  of  the  visual  targets  while  in  transit.  Performance  while  in  transit  did  not  differ 
between  the  MS-Present  versus  the  No-MS  conditions.  In  contrast,  the  MCs  detected  fewer 
visual  targets  (average  of  56%)  while  in  an  observation/contact  phase.  Moreover,  there  was  a 
difference  across  conditions  where  significantly  fewer  visual  targets  were  detected  in  the  No-MS 
than  in  the  MS-Present  condition.  This  result  shows  that  the  MC  had  less  visual  attention  to 
allocate  to  the  target  detection  task  in  the  No-MS  condition  than  in  the  MS-Present  condition. 


Figure  4.17 


Percent  Detection  of  Visual  Targets 
by  the  Mssion  Commander 


□  NdMS 

□  MSR-esent 


Mission  Artivity 
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4.4  Post-mission  comparison  ratings 

At  the  end  of  the  each  day  of  testing,  participants  completed  a  questionnaire  to  directly  compare 
the  MS-Present  versus  the  No-MS  conditions  for  both  the  FP  and  for  the  MC  roles.  The  post¬ 
mission  comparison  questionnaire  is  shown  in  Annex  F  and  the  corresponding  average  ratings 
are  shown  in  Annex  G. 

4.4.1  Flying  pilot  role 

Figure  4.18  summarizes  the  comparison  ratings  for  the  FP  role.  In  this  Figure,  a  value  of  “4” 
would  indicate  that  there  is  no  rated  difference  between  having  a  MS  present  versus  having  no 
MS.  As  seen  in  Figure  4. 18,  all  of  the  comparison  values  are  slightly  less  than  4.  This  shows 
that  for  the  FP  role,  participants  rated  their  performances  and  experience  as  a  MC  as  slightly 
better  when  a  MS  was  present  as  compared  to  when  no  MS  was  present. 

Figure  4.18 


Comparison  Ratings 
Flying  Pilot  Role 
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Rating  <  4  better  with  MS  Present 
Rating  >  4  better  with  NO  MS 

For  the  Flying  Pilot  role,  ratings  were  generally 
better  when  Mission  Specialist  was  present 
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4.4.2  Mission  commander  role 

Figure  4.19  summarizes  the  comparison  ratings  for  the  MC  role.  A  value  of  “4”  would  indicate 
that  there  is  no  rated  difference  between  having  a  MS  present  versus  having  no  MS.  All  of  the 
comparison  values  are  less  than  4  with  many  reaching  a  value  of  2  or  less.  This  shows  that  for 
the  MC  role,  participants  rated  their  performances  and  experience  as  an  MC  as  better  when  a  MS 
was  present  as  compared  to  when  no  MS  was  present. 

Figure  4.19 
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5  Conclusions 

In  this  experiment,  the  CSE  framework  developed  at  the  beginning  of  the  TAMSS  SA  project 
was  used  to  examine  the  question  of  how  the  presence  of  a  mission  specialist  would  affect  the 
situation  awareness,  performance,  and  workload  of  the  CH-146  aircrew  during  recce  missions. 
The  advantage  of  using  the  CSE  framework  is  that  it  provided  a  comprehensive  set  of  measures 
that  allowed  a  broad  overview  of  how  the  mission  specialist  affected  the  task.  Furthermore,  the 
use  of  both  subjective  and  objective  measures  of  SA,  workload,  and  perfonnance  allowed 
confirmation  across  the  various  aspects  of  the  situation. 

When  participants  took  the  role  of  the  flying  pilot,  the  presence  of  the  mission  specialist  had  a 
moderate  effect  on  performance.  Subjectively,  the  FPs  felt  that  their  perfonnance  was  somewhat 
better  when  a  mission  specialist  was  present,  particularly  in  the  observation  and  recce  phases  of 
the  mission.  There  was  little  evidence  for  an  influence  of  the  mission  specialist  on  situation 
awareness  or  workload  for  the  FPs.  Because  the  focus  in  this  experiment  was  on  the  mission 
commander,  no  objective  measures  of  performance  were  collected. 

When  participants  took  the  role  of  the  mission  commander,  they  reported  substantial  effects  on 
performance,  workload,  and  SA.  In  general,  the  MC  found  it  easier  to  interact  with  the  ERSTA 
system  when  a  mission  specialist  was  present.  The  reduced  workload  generalized  to  using  the 
radio  coinms  suggesting  that  the  MC  felt  generally  overloaded  when  the  task  of  operating  the 
sensor  was  added  to  all  of  the  other  demands  of  the  missions.  As  in  the  FP  role,  the  MCs  felt 
that  their  workload  was  increased  in  the  observation  and  recce  phases  of  the  missions.  The 
results  of  the  objective  measurements  supported  the  subjective  results.  In  the  presence  of  the 
mission  specialist,  the  MC  showed  a  greater  amount  of  heads-up  time,  looked  at  the  sensor 
image  less,  and  the  sensor  itself  was  moved  less.  Objective  workload  was  also  greater  for  the 
MC  when  he  or  she  was  operating  the  sensor.  Substantially  fewer  targets  were  detected, 
especially  in  the  portions  of  the  mission  (observations  and  in  contacts  in  recce  zone)  that 
required  the  MC  to  be  processing  other  information  or  interacting  with  other  aspects  of  the 
technology. 
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In  summary,  these  results  provide  a  clear  picture  of  how  the  presence  of  a  mission  specialist 
enhanced  the  subjective  and  objective  task  performance.  The  mission  commander,  freed  from 
the  increased  demands  of  operating  the  sensor,  had  more  mental  attention  to  put  towards  the 
primary  demands  of  the  MC  role. 

Experiment  3  was  also  technically  progressive.  The  ERTA-like  model  was  successfully 
extended  to  provide  the  requisite  level  of  functionality  thereby  enabling  the  aircrew  to  use  the 
digital  moving  map  and  the  sensor  capabilities  in  a  realistic  and  appropriate  manner.  The 
simulation  environment  was  robust  and  stable.  The  data  collection  utility  was  stable  and 
accurate  throughout  the  experiment  and  the  HLA-based  distributed  simulation  which  the  CACR 
used  to  connect  the  model  of  the  ERSTA  system  to  the  CACR  CHI 46  flight  simulator  ran 
flawlessly  for  a  minimum  of  eight-to-ten  hours  per  day  across  twelve  days  of  testing. 

The  finding  that  the  impact  of  cockpit  technologies  (e.g.,  digital  map,  sensor)  on  crew 
performance,  situation  awareness,  and  workload  can  be  systematically  measured  allows  for  three 
important  conclusions.  First,  it  is  concluded  that  the  CSE  framework  is  a  useful  and  workable 
framework  for  M&S  programs.  Second,  it  is  concluded  that  the  impact  of  new  aircraft 
technologies  on  aircrew  can  be  systematically  and  meaningfully  measured  using  the  level  of 
system  fidelity  that  is  represented  in  the  CACR  simulation  environment.  Third,  it  is  concluded 
that  the  CSE  framework  and  the  simulation  environment  that  was  developed  in  the  TAMSS  SA 
project  can  be  used  to  affect  the  design,  prototype,  test,  build  and  implementation  processes  in 
simulation-based  acquisition  programs. 
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ANNEX  A 
Mission  Scenarios 
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TI-02  TAMSS  SA  -  EXPERIMENT  #3  -  ZONE  RECCE  #1 


Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  10  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN 
troops  out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this 
territory  for  the  past  14  months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive 
them  out  of  the  territory.  Recently,  in  the  past  four  months,  the  attacks  have  increased  in  number  and  in 
ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD) 
with  the  main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to 
live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having 
been  occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to 
continue  the  push  to  the  St  John  River.  These  key  areas  are  the  bridges  crossing  the  Oromocto,  Nerepis 
and  Otnabog  Rivers  and  choke  points  out  of  the  hilled  areas  on  the  West  side  of  the  St  John  river  leading 
to  the  main  river  crossing  areas  used  by  the  FDN  for  their  initial  invasion  some  14  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen 
Stolitchnoya  (callsign  Stiletto)  from  Lithuania.  The  main  UN  offensive  has  pushed  the  main  FDN  forces 
out  of  the  territory.  However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to 
disrupt  the  UN  operation  and  slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup 
and  attempt  to  re-start  their  guerrilla  activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  GE  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (Ml 09)  and  Armd  (Leo  2)  from  the  German  5  Spa-Panzer  Div  (SPD) 

•  UK  Inf  Bn  (3  Bn  Royal  Grenadiers) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AWACs  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (CF18s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  45  km  West  of  the  objective  area,  at  the  Brockway  Airfield 
(FL  480485). 
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The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These 
include: 


bridges  at: 


Bridge  Name 

GL  105558 

100 

GL  097533 

101 

GL  091499 

102 

GL  063467 

103 

GL  098450 

104 

points  at: 

Choke  Point  Name 

GL  187647 

200 

GL  205578 

201 

GL  190468 

202 

GL  211431 

203 

You  are  a  Griffon  crew  assigned  to  conduct  Recce  tasks  with  the  newly  equipped  ERSTA  system.  The 
Comd  UNMNB  has  been  impressed  with  the  support  provided  to  date  since  the  ERSTA  is  the  best  long 
range  rapidly  deployable  Recce  system  available  in  the  Bde.  The  ERSTA  equipped  Griffons  played  a  key 
roll  in  aiding  the  rapid  advance  of  the  MNB  in  the  past  9  days. 

It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant 
importance.  The  mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the 
territory  may  be  halted  as  a  result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  bridge  at  GL  091499.  The  last  report  from  the  patrol 
indicated  that  they  had  been  fired  at  sporadically  from  the  surrounding  hills  to  the  East  of  their  positions. 

It  is  extremely  important  that  the  HQ  determine  the  status  of  the  bridge  and  if  the  FDN  are  attempting  to 
mine  or  destroy  the  bridge. 
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Pre  Fit  Brief  to  Crew  from  Ops 


Situation: 

General: 

•  as  per  intro 
En: 

•  likely  en  activity  in  area  of  GL  091499  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  GL  1856  -  6  hrs  ago 

•  intsum  -  bypassed  en  units  (Platoon  -)  East  of  20  Easting 

•  contact  report  -  en  patrol  raided  village  in  area  of  GL  0449  -  12  hrs  ago, 

appeared  to  be  headed  East 

Fr: 

•  Recce  Tp  located  in  area  GL  0555  (6  x  HUM  TOW) 

•  Arty  Bty  located  at  FL  998504  (8  x  M109-A6) 

•  Inf  located  at  GL  059531  (camouflaged  tents) 

.  Patrol  on  bridge  at  GL  082519  (3  x  HUM  AVENGER  +  1  x  M2A3  BRADLEY) 

•  attached  for  duration  of  mission  -  W  Bty  through  G24 

•  CAS  on  call  from  MAGIC  (10  min.  notice  required) 

Wx: 

•  5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 

•  sunrise:  1200Z,  sunset:  0300Z 

Airspace: 

•  restrictions  as  per  ACO 


Mission: 


Gain  observation  on  the  bridge  at  GL  091499  to  determine  if  the  bridge  is  intact  and  observe 
the  bridge  until  T43B  elements  arrive. 

Execution: 


General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

1 .  gain  observation  onto  the  bridge 

2.  determine  if  the  bridge  is  intact 

3.  maintain  observation  until  T43B  arrives 

Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  arty  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 
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AD  Assets: 

.  1  x  M2A3  BRADLEY  is  located  at  GL  141579  (5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 


Service  Support: 


.  FARP  at  GL  027481  (open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 
.  FSH  located  at  FL  884566 

•  perseqpt-SOP 

Command  and  Signals: 


Command: 

.  UC  408  Sqn  B  Fit 
.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  SPYGLASS 

•  objective  intact  MANHOLE 

•  objective  destroyed  COLDSTART 

COMMs: 

•  as  per  CEOIs 


38 


TAMSS  SA  Experiment  3 


Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

TWEED 

FL  560565 

<500’ 

0930-1000 

BLISSVILLE 

FL  912532 

FINNEGANS 

FL  995577 

RP 

CLONES 

GL  057552 

<250’ 

N/A 

Egress 

PETERSVILLE 

GL  046498 

<  500’ 

1000-1100 

BLISSVILLE 

FL  912532 

TWEED 

FL  560565 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted  from  other 
friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The  Restricted 
Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  GL  106442 

.  GL  077450 

.  GL  095575 

.  GL  131563 

There  is  an  active  LLTR  immediately  South  of  the  objective  bridge,  passing  immediately  above  the  ROZ 
(250’-500’),  one  way,  headed  SW.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the  LLTR 
on  egress  from  BAI  missions  in  FDN  held  territory  East  of  the  St  John  River.  The  LLTR  is  aligned  with: 

.  GL  2756  and  FL  9039 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or 
release  point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  FIR 

FL  850550,  FL  960613,  GL  040633,  FL  222655 

PL  OAK 

FL  850440,  GL  046498,  GL  256575 

PLASH 

FL  850330,  GL  078371,  GL  269496 
PL  BASS 

FL  956690,  FL  910623,  FL  897580,  FL  924465,  FL  992405,  FL  960330 
PL  DOG 

FL  965686,  FL  967640,  GL  006553,  GL  070435,  GL  087412,  GL  079369,  GL  117338 
PL  HORSE 

GL  000720,  GL  037662,  GL  059588,  GL  124550,  GL  1 14522,  GL  172502,  GL  247389 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AWAC 

MAGIC 

HQII  /  A19.225 

No 

Bde  HQ 

92 

34.50 

Fill  5 

Patrol 

I23A 

33.90 

Fill  4 

Arty 

G24 

35.20 

Fill  3 

Recce  Tp 

T43B 

42.65 

Fill  1 

Fit  Ops 

0 

49.90 

Fill  2 

FARP 

52C 

46.50 

Fill  2 

CAS  (on  call) 

KUGAR 

HQII  /  A03.625 

No 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

02 

0930-1000 

73 

1000-1030 

61 

1030-1100 

51 

1100-1130 

40 

1130-1200 

22 

Mode  2: 

N/A 

1324 

Mode  3: 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination  Point,  at 
an  altitude  of  400  feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point.  Once  the  aircrew  is 
comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 


Experimenter  Inputs  (NOT  FOR  EXPERT  USER  CONSUMPTIONS 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of 
the  synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

.  5  x  Ml  13  APCs  will  transit  NW  on  the  black  track  from  GL  050553  to  GL  023569.  The  APCs 

need  to  be  moving  as  the  helicopter  is  transiting  between  FINNEGAN  and  CLONES. 

•  A  burning/crashed  AH-64  Apache  will  be  on  the  ground  at  GL  070554. 

•  4  x  BMP  Armoured  Fighting  Vehicles  will  transit  North  on  the  road  from  GL  092470  to  GL  114522. 
The  BMPs  need  to  be  moving  as  the  Griffon  reaches  CLONES. 
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TAMSS  SA  -  EXPERIMENT  #3  -  ZONE  RECCE  #2 


Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  1 0  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN 
troops  out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this 
territory  for  the  past  12  months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive 
them  out  of  the  territory.  Recently,  in  the  past  three  months,  the  attacks  have  increased  in  number  and  in 
ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD) 
with  the  main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to 
live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having 
been  occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to 
continue  the  push  to  the  South  and  West  of  the  St  John  River.  These  key  areas  are  the  bridges  crossing 
the  Nerepis  River  and  choke  points  along  the  road  paralleling  the  NW-SE  rail  line  and  extending  South  to 
the  St.  John  River.  Wellsford  was  a  main  crossing  area  used  by  the  FDN  during  their  initial  incursion 
some  12  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen 
Ruberg  (callsign  Thunder)  from  Lithuania.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out 
of  the  territory.  However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to 
disrupt  the  UN  operation  and  slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup 
and  attempt  to  re-start  their  guerrilla  activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  US  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (Ml 09)  and  Armd  (ABRAMS)  from  the  1st  US  Armd  Div 

•  UK  Inf  Bn  (3  Bn  Royal  Grenadiers) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AWAC  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (CF18s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  40  km  North  of  the  objective  area,  at  the  Hersey  Corner 
Airstrip  (GL  100792). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These 
include: 


bridges  at: 

.  GL  091499 


Bridge  Name 
105 
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.  GL  093466  106 

.  GL  099450  107 

.  GL  088433  109 

.  GL  089415  110 

choke  points  at: 

Choke  Point  Name 

.  GL  079370  204 

.  GL  017395  205 

.  FL  993403  206 

.  GL  226436  207 


It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant 
importance.  The  mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the 
territory  may  be  halted  as  a  result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  choke  point  at  GL  079370.  The  last  report  from  the 
patrol  indicated  that  they  had  observed  increased  vehicular  traffic  and  massing  of  Armd  unit(s)  to  the 
South  of  their  objective.  It  is  extremely  important  that  the  HQ  determine  the  status  of  the  choke  point  and 
if  the  FDN  are  attempting  to  advance  North. 
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Pre  Fit  Brief  to  Crew  from  Ops 


Situation: 

General: 

•  as  per  intro 
En: 

•  en  activity  in  area  of  GL  079370  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  GL  193431  -  3  hrs  ago 

•  contact  report  -  en  patrol  raided  village  in  area  of  GL  2243  -  8  hrs  ago,  appeared 
to  be  headed  West 

Fr: 

•  Recce  Tp  located  in  area  GL  0345  (6  x  HUM  TOW) 

.  Arty  Bty  located  at  FL  050503  (8  x  M 1 09-A6) 

•  Inf  Platoon  located  at  GL  107440  (camouflaged  tents) 

.  Patrol  on  bridge  at  GL  088433  (4  x  M2A3  BRADLEY) 

•  attached  for  duration  of  mission  -  W  Bty  through  G24 

•  CAS  on  call  from  MAGIC  (10  min.  notice  required) 

Wx: 

•  5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 

•  sunrise:  1200Z,  sunset:  0300Z 

Airspace: 

•  restrictions  as  per  ACO 


Mission: 


Gain  observation  on  the  choke  point  at  GL  079370  (objective)  to  determine  en  activity  and 
continue  to  observe  until  T43B  elements  arrive. 

Execution: 

General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

4.  gain  observation  onto  the  choke  point 

5.  determine  if  there  is  any  en  advance  to  /  beyond  the  choke  point 

6.  maintain  observation  until  T43B  arrives 


Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  arty  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 
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AD  Assets: 

•  1  x  M2A3  BRADLEY  is  located  at  FL  954438  (5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 


Service  Support: 


.  FARPatGL  141461  (open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 

•  FSH  located  at  GL  098630 

•  perseqpt-SOP 

Command  and  Signals: 


Command: 

.  UC  408  Sqn  B  Fit 
.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  TELEPHOTO 

•  objective  secure  from  en  LOCKDOWN 

•  en  movement  North  of  objective  GALLOP 

COMMs: 

•  as  per  CEOIs 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

ROCKWELL  STREAM  BRIDGE 

GL  013734 

<  500’ 

0930-1000 

KNOWLTON  HILL 

GL  060587 

BIG  BOG 

FL  997517 

RP 

RODDYS  LAKE 

GL  043443 

<250’ 

N/A 

Egress 

LYON  BRIDGE  ROAD 

GL  096451 

<  500’ 

1000-1100 

KNOWLTON  HILL 

GL  060587 

ROCKWELL  STREAM  BRIDGE 

GL  013734 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted  from  other 
friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The  Restricted 
Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  GL  074333 

.  FL  985393 

.  FL  976430 

.  GL  135366 

There  is  an  active  LLTR  immediately  West  of  the  objective  choke  point,  passing  immediately  above  the 
ROZ  (250-500’),  one  way,  headed  NNE.  It  is  expected  that  some  friendly  fighter  activity  may  be  using 
the  LLTR  on  egress  from  BAI  missions  in  FDN  held  territory  SW  of  the  St  John  River.  The  LLTR  is 
aligned  with: 


.  FL  9926  and  FL  209778 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or 
release  point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  TROUT 

FL  890720,  FL  964686,  GL  107728,  GL  210779 
PL  PERCH 

FL  909501,  GL  060587,  GL  173569,  GL  253590 
PL  BASS 

FL  900320,  GL  017395,  GL  096451,  GL  260486 
PL  LION 

FL  931738,  FL  898560,  GL  010310 

PL  TIGER 

GL  237383,  GL  167607,  GL  138797 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AWAC 

MAGIC 

HQII  /  A89.625 

No 

Bde  HQ 

92 

46.25 

Fill  5 

Patrol 

122  A 

39.80 

Fill  4 

Arty 

G24 

34.50 

Fill  3 

Recce  Tp 

T43B 

48.65 

Fill  2 

Fit  Ops 

0 

49.90 

Fill  1 

FARP 

52  D 

46.50 

Fill  1 

CAS  (on  call) 

HAWK 

HQII  /A67. 125 

No 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

73 

0930-1000 

61 

1000-1030 

53 

1030-1100 

41 

1100-1130 

10 

1130-1200 

70 

Mode  2: 

N/A 

1324 

Mode  3: 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination  Point,  at 
an  altitude  of  400  feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point.  Once  the  aircrew  is 
comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 


Experimenter  Inputs  (NOT  FOR  EXPERT  USER  CONSUMPTIONS 


These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of 
the  synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

•  6  x  M1A2  Abrams  MBTs  will  transit  WSW  on  the  road  from  GL  040497  to  FL  991481.  The 
Abrams  need  to  be  moving  as  the  helicopter  is  transiting  to,  but  short  of  BIG  BOG. 

•  A  burning  M939A2  5-Ton  Truck  will  be  on  the  ground  at  GL  067451 . 

•  6  x  BTR  80s  will  transit  North  on  the  road  from  GL  1 00336  to  GL  079368.  The  BTRs  need  to  be 
moving  as  the  Griffon  reaches  RODDYS  LAKE. 
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TAMSS  SA  -  EXPERIMENT  #3  -  ZONE  RECCE  #3 


Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  10  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN 
troops  out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this 
territory  for  the  past  10  months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive 
them  out  of  the  territory.  Recently,  in  the  past  two  months,  the  attacks  have  increased  in  number  and  in 
ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD) 
with  the  main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to 
live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having 
been  occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to 
continue  the  push  to  the  North  and  West  of  the  St  John  River.  These  key  areas  are  the  bridges  West  of 
Otnabog  Lake  and  choke  points  on  the  West  side  of  the  St  John  river,  used  by  the  FDN  for  their  initial 
incursion  some  10  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen 
Stanlowski  (callsign  Grimace)  from  Poland.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out 
of  the  territory.  However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to 
disrupt  the  UN  operation  and  slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup 
and  attempt  to  re-start  their  guerrilla  activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  GE  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (MLRS)  and  Armd  (Leo  2)  from  the  German  5  Spa-Panzer  Div  (SPD) 

.  UK  Arty  Bn  (21st  FA) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AWAC  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (F18s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  35  km  South  West  of  Black  Clarendon  (FL  8521 ). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These 
include: 


bridges  at: 
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Bridge  Name 


GL  170651  111 

GL 187647  112 

GL 196652  113 

GL 207649  114 


choke  points  at: 

.  GL  204704 

.  GL  174710 

.  GL  139640 

.  GL  085735 

.  GL  041633 

.  GL  197657 


Choke  Point  Name 
208 

209 

210 
211 
212 
213 


You  are  a  Griffon  crew  assigned  to  conduct  Recce  tasks  with  the  newly  equipped  ERSTA  system.  The 
Comd  UNMNB  has  been  impressed  with  the  support  provided  to  date  since  the  ERSTA  is  the  best  long 
range  rapidly  deployable  Recce  system  available  in  the  Bde.  The  ERSTA  equipped  Griffons  played  a  key 
roll  in  aiding  the  rapid  advance  of  the  MNB  in  the  past  7  days. 

It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant 
importance.  The  mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the 
territory  may  be  halted  as  a  result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  bridges  leading  to  the  choke  point  /  objective  area  at 
GL  197657.  The  last  report  from  the  patrol  indicated  that  they  had  taken  mortar  fire  at  sporadically  from 
the  North  and  West  of  their  positions.  It  is  extremely  important  that  the  HQ  determine  the  status  of  the 
bridges  and  the  choke  point,  and  if  the  FDN  are  attempting  to  mine  or  destroy  the  bridges  and  /or 
advance  South  through  the  choke  point. 
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Pre  Fit  Brief  to  Crew  from  Ops 


Situation: 

General: 

•  as  per  intro 
En: 

•  likely  en  activity  in  area  of  GL  197657  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  GL  1774  -  8  hrs  ago 

•  intsum  -  ZSU-234  activity  at  Tantawanta  Bridge  -  4  hrs  ago 

•  contact  report  -  en  patrol  raided  village  in  area  of  Fentons  -  6  hrs  ago,  appeared 
to  be  headed  North  West 

Fr: 

•  Recce  Tp  located  in  area  GL  1472  (6  x  HUM  TOW) 

.  MLRS  Bty  located  at  GL  135585  (6  x  M270  MLRS) 

•  Inf  located  at  GL  210610  (camouflaged  tents) 

•  Patrol  at  road  intersection  GL  178629  (4  x  HUM  AVENGER) 

•  attached  for  duration  of  mission  -  W  Bty  through  G24 

•  CAS  on  call  from  MAGIC  (10  min.  notice  required) 

Wx: 

•  5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 

•  sunrise:  1200Z,  sunset:  0300Z 

Airspace: 

•  restrictions  as  per  ACO 


Mission: 


Gain  observation  on  the  key  bridges,  determine  if  intact  and  observe  the  choke  point  at  GL 
197657  until  T43B  elements  arrive. 

Execution: 

General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

7.  gain  observation  onto  the  key  bridges 

8.  determine  if  the  bridges  are  intact 

9.  maintain  observation  on  the  choke  point  until  T43B  arrives 
Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  MLRS  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 
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AD  Assets: 

•  1  x  M2A3  BRADLEY  is  located  at  GL  249634(5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 


Service  Support: 


.  FARP  at  GL  068548  (open  1000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 
.  FSH  located  at  GL  023485 

•  perseqpt-SOP 

Command  and  Signals: 


Command: 

.  UC  408  Sqn  B  Fit 

.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

WINDSCREEN 
FANCY 
BITTER 
LOCOMOTIVE 

COMMs: 

•  as  per  CEOIs 


objective  under  observation 
all  bridges  intact 
any  bridge  destroyed 
en  movement  South  of  choke  point 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

BLACK  CLAREDON 

FL  902340 

<  500’ 

0930-1000 

WELLSFORD 

GL  078370 

BELL  BRIDGE  RUIN 

GL  097496 

RP 

DAY  HILL 

GL  193545 

<250’ 

N/A 

Egress 

MALLORY-KERR  ROADS 

GL  106578 

<  500’ 

1000-1100 

WELLSFORD 

GL  078370 

BLACK  CLAREDON 

FL  902340 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted  from  other 
friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The  Restricted 
Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  GL  120627 

.  GL  114683 

.  GL  232684 

.  GL  212622 

There  is  an  active  LLTR  immediately  West  of  the  objective  bridge,  passing  immediately  above  the  ROZ 
(250’-500’),  one  way,  headed  SSE.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the 
LLTR  on  egress  from  BAI  missions  in  FDN  held  territory  North  of  the  St  John  River.  The  LLTR  is  aligned 
with: 

.  GL  127820  and  GL  200320 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or 
release  point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  RAM 

FL  891789,  FL  992740,  GL  150734,  FL  262669 
PL  STEER 

FL  903627,  GL  023570,  GL  193502,  GL  262477 
PL  HOG 

FL  89841 9,  FL  947442,  GL  078370,  GL  230337 
PL  ROD 

FL  992740,  GL  041633,  GL  048500,  GL  000333 

PL  REEL 

GL  243792,  GL  262669,  GL  262447,  GL  230337 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AWAC 

MAGIC 

HQII  /  A73.925 

No 

Bde  HQ 

92 

36.25 

Fill  5 

Patrol 

122  A 

59.80 

Fill  3 

Arty 

G24 

33.90 

Fill  4 

Recce  Tp 

T43B 

42.65 

Fill  1 

Fit  Ops 

0 

49.90 

Fill  2 

FARP 

52  D 

46.50 

Fill  2 

CAS  (on  call) 

TIGER 

HQII  /A48.625 

No 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

03 

0930-1000 

51 

1000-1030 

73 

1030-1100 

41 

1100-1130 

30 

1130-1200 

00 

Mode  2: 

N/A 

1324 

Mode  3: 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination  Point,  at 
an  altitude  of  400  feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point.  Once  the  aircrew  is 
comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 


Experimenter  Inputs  (NOT  FOR  EXPERT  USER  CONSUMPTIONS 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of 
the  synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

.  6  x  Ml  10  SP  Howitzers  will  transit  NNE  on  the  road  from  GL  154508  to  GL  169537.  The  Ml  10s 

need  to  be  moving  as  the  helicopter  is  transiting  between  BELL  BRIDGE  RUIN  and  DAY  HILL. 

•  A  landed  MI-28  Havoc  will  be  on  the  ground  at  GL  209629. 

•  4  x  T-72  MBTs  will  transit  South  on  the  road  from  GL  203703  to  the  choke  point  /  objective  at  GL 
197657.  The  MBTs  need  to  be  moving  as  the  Griffon  reaches  DAY  HILL. 
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TAMSS  SA  -  EXPERIMENT  #3  -  ZONE  RECCE  #4 


Tactical  Scenario  -  Peace  Support  Operations 

It  is  day  1 0  of  Op  NOMAD.  Op  NOMAD  is  the  UN  counteroffensive  to  push  the  final  elements  of  the  FDN 
troops  out  of  the  territory  formerly  known  as  Gagetownia.  The  FDN  troops  have  been  invading  this 
territory  for  the  past  8  months,  conducting  sporadic  attacks  on  innocent  civilians  in  an  attempt  to  drive 
them  out  of  the  territory.  Recently,  in  the  past  month,  the  attacks  have  increased  in  number  and  in 
ferocity. 

A  number  of  emergency  UN  meetings  have  occurred  resulting  in  a  UN  Chapter  6  operation  (Op  NOMAD) 
with  the  main  objective  to  force  the  FDN  troops  out  of  the  territory  and  allow  the  local  Gagetownians  to 
live  in  peace. 

The  FDN  are  a  well  equipped  para-military  organization  who  have  been  trained  by  Soviet  forces,  having 
been  occupied  as  a  soviet  state  for  many  years.  Their  equipment  is  predominantly  Soviet  based. 

Future  MNB  operations  require  a  number  of  critical  areas  to  be  maintained  intact  for  the  UN  troops  to 
continue  the  push  West  and  North  of  the  St  John  River.  These  key  areas  are  the  bridges  and  access 
routes  crossing  the  Oromocto  and  St.  John  (West  of  Gagetown)  Rivers  leading  to  the  main  incursion 
points  used  by  the  FDN  for  their  initial  invasion  some  8  months  ago. 

The  UN  force  is  being  led  by  the  UN  Multi-National  Brigade  (UNMNB)  under  the  command  of  BGen 
Leboeuf  (callsign  Roaster)  from  France.  The  main  UN  offensive  has  pushed  the  main  FDN  forces  out  of 
the  territory.  However,  it  is  expected  that  a  number  of  layback  patrols  have  remained  in  the  area  to 
disrupt  the  UN  operation  and  slow  down  the  advance.  This  will  allow  time  for  the  FDN  forces  to  regroup 
and  attempt  to  re-start  their  guerrilla  activities. 

Canada  has  provided  the  following  forces: 

•  a  Mech  Inf  Coy  (A  Coy  2PPCLI)  attached  to  a  UK  Inf  Bn 

•  a  Recce  Tp  (A  Tp  LdSH)  attached  to  a  GE  armoured  unit 

•  a  helicopter  Fit  (B  Fit  408  Sqn)  assigned  to  MNB  HQ 

•  fighter  assets  from  4  Wing  Cold  Lake 

Other  UN  assets  of  importance  include: 

•  Arty  (Ml 09)  and  Armd  (Leo  2)  from  the  German  5  Spa-Panzer  Div  (SPD) 

.  UK  Arty  Bn  (21st  FA) 

.  US  Inf  Bn  (2  Bn  1  ID) 

•  AWAC  support 

•  CAS  (GR8s)  from  RAF  Wittering,  deployed  to  an  undisclosed  5  ATAF  field  location 

•  CAS  /  CAP  support  (F18s)  from  4  Wing,  deployed  to  an  undisclosed  5  ATAF  DOB 

Currently,  the  helicopter  unit  is  located  some  40  km  South  East  of  the  objective  area,  at  the  Blue 
Mountain  Correctional  Facility  (GL  223453). 

The  MNB  has  secured  (albeit  loosely)  a  number  key  areas,  important  to  the  Op  NOMAD  advance.  These 
include: 


bridges  at: 
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Bridge 

.  FL  904710  115 

.  FL  926797  116 

.  FL  929804  117 

.  FL  978822  118 

.  FL  960725  119 

.  FL  970777  120 

.  FL  996724  121 

choke  points  at: 

Choke 

.  FL  988780  214 

.  FL  978775  215 

.  FL  963752  216 

.  FL  996717  217 


Name 


Point  Name 


You  are  a  Griffon  crew  assigned  to  conduct  Recce  tasks  with  the  newly  equipped  ERSTA  system.  The 
Comd  UNMNB  has  been  impressed  with  the  support  provided  to  date  since  the  ERSTA  is  the  best  long 
range  rapidly  deployable  Recce  system  available  in  the  Bde.  The  ERSTA  equipped  Griffons  played  a  key 
roll  in  aiding  the  rapid  advance  of  the  MNB  in  the  past  6  days. 

It  is  0830  and  you  have  been  informed  that  you  are  required  to  complete  a  mission  of  significant 
importance.  The  mission  needs  to  be  done  quickly  since  the  momentum  of  the  MNB  push  through  the 
territory  may  be  halted  as  a  result  of  possible  FDN  activity. 

The  HQ  has  lost  contact  with  a  patrol  securing  the  bridge  at  FL  960725.  The  last  report  from  the  patrol 
indicated  that  they  had  been  fired  at  sporadically  from  the  buildings  to  the  South  West  of  their  positions.  It 
is  extremely  important  that  the  HQ  determine  the  status  of  the  bridge  and  if  the  FDN  are  attempting  to 
mine  or  destroy  the  bridge. 


57 


TAMSS  SA  Experiment  3 


Pre  Fit  Brief  to  Crew  from  Ops 


Situation: 

General: 

•  as  per  intro 
En: 

•  likely  en  activity  in  area  of  FL  960725  (objective)  (last  contact  by  patrol  in  area) 

•  contract  report  -  sniper  activity  in  area  FL  8980  -  4  hrs  ago 

•  intsum  -  2S6  activity  in  the  area  of  Wood  Meadow  -  6  hrs  ago 

•  contact  report  -  en  patrol  raided  village  in  area  of  Lower  Lincoln  -  6  hrs  ago, 

appeared  to  be  headed  North  West 

Fr: 

•  Recce  Tp  located  in  area  FL  9866  (6  x  HUM  TOW) 

.  MLRS  Bty  located  at  FL  982652  (6  x  M270  MLRS) 

•  Inf  located  at  FL  994770  (camouflaged  tents) 

.  Patrol  on  bridge  at  FL  996724  (1  x  M2A3  BRADLEY  +  3  x  HUM  AVENGER) 

•  attached  for  duration  of  mission  -  W  Bty  through  G24 

•  CAS  on  call  from  MAGIC  (10  min.  notice  required) 

Wx: 

•  5000  OVC,  4  NM  vis,  localized  drizzle,  temp  15°,  dewpoint  10°,  pressure  29.95” 

•  sunrise:  1200Z,  sunset:  0300Z 

Airspace: 

•  restrictions  as  per  ACO 


Mission: 


Gain  observation  on  the  bridge  at  FL  960725  to  determine  if  the  bridge  is  intact  and  observe 
the  bridge  until  T43B  elements  arrive. 

Execution: 


General  Outline: 

•  In  line  with  the  mission,  you  have  three  objectives: 

10.  gain  observation  onto  the  bridge 

11.  determine  if  the  bridge  is  intact 

12.  maintain  observation  until  T43B  arrives 

Groupings  and  Tasks: 

•  You  will  be  the  sole  CH146  equipped  with  ERSTA  on  the  mission 
Fire  Support: 

•  arty  is  available  on  call  throughout  the  mission  through  G24 

•  CAS  is  available  through  MAGIC  on  10  min  notice 


58 


TAMSS  SA  Experiment  3 


AD  Assets: 

•  1  x  M2A3  BRADLEY  is  located  at  FL  960650  (5  km  Radius  down  to  50’) 

Co-ord  Inst: 

•  timings:  depart  on  mission  NLT  0930 

•  route:  as  per  ACO 

•  upon  passing  RP  contact  Bde  HQ  (C/S  92)  and  advise  of  ETA  on  observation  area 

•  maintain  contact  with  92  through-out  Recce  phase 


Service  Support: 


.  FARP  at  GL  098722  (open  1 000-1030)  -  fuel  only  (HEMTT  Fuel  Bowser) 
.  FSH  located  at  FL  195545 

•  perseqpt-SOP 

Command  and  Signals: 


Command: 

.  UC  408  Sqn  B  Fit 
.  TACCON  Bde  HQ  (C/S  92) 

Codewords: 

•  objective  under  observation  ZOOM 

•  objective  intact  HARVEST 

•  objective  destroyed  RECOIL 

COMMs: 

•  as  per  CEOIs 
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Airspace  Control  Orders 

A  special  helicopter  corridor  has  been  reserved  for  this  mission: 


Co-ord  Point 

Location 

Altitude 

Time  (Z) 

Ingress 

CENTRAL  HAMPSTEAD 

GL  250590 

<500’ 

0930-1000 

LAWFIELD-BOUNDARY  ROADS 

GL  139640 

NW  KNOWLTON  HILL 

GL  048609 

RP 

BROAD  ROAD  CLEARING 

FL  984636 

<250’ 

N/A 

Egress 

LAUVINA  ROAD  CLEARING 

GL  006703 

<500’ 

1000-1100 

LAWFIELD-BOUNDARY  ROADS 

GL  139640 

CENTRAL  HAMPSTEAD 

GL  250590 

The  area  in  the  vicinity  of  the  objectives  has  been  co-ordinated  with  MNB  HQ  to  be  restricted  from  other 
friendly  helicopter  activity  except  for  emergency  operations  and  enemy  contact.  The  Restricted 
Operations  Zone  (ROZ)  is  capped  at  250’  and  is  bounded  by: 

.  FL  920678 

.  FL  936792 

.  FL  975782 

.  FL  973700 

.  FL  945672 

There  is  an  active  one  way  LLTR  South  of  the  objective  bridge,  headed  ENE  and  passing  immediately 
South  of  the  ROZ  (SFC  to  <  500’.  It  is  expected  that  some  friendly  fighter  activity  may  be  using  the  LLTR 
on  egress  from  BAI  missions  in  FDN  held  territory  West  of  the  objective.  The  LLTR  is  aligned  with: 

.  FL  8564  and  GL  2575 

All  aircraft  activity  in  the  territory  is  co-ordinated  with  AWAC  (Magic).  Upon  passing  any  co-ord  point  or 
release  point  a  call  is  to  be  made  indicating  your  callsign  and  the  point  you  passing. 

PL  SLEET 

FL  920832,  GL  094835,  GL  195794,  GL  262669 

PL  HAIL 

FL  901620,  GL  171502,  GL  249369 
PL  WAVE 

FL  96041 0,  FL  908510,  FL  897561 ,  FL  920832 

PL  RIPPLE 

GL  070305,  GL  097496,  GL  141579,  GL  139640,  GL  243793 
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Communications  Electronic  Operating  Instructions: 


COMMs: 


Unit 

C/S 

Freq 

Crypto 

AWAC 

MAGIC 

HQII  /All. 125 

No 

Bde  HQ 

92 

37.85 

Fill  5 

Patrol 

I12A 

58.50 

Fill  1 

Arty 

G24 

34.50 

Fill  3 

Recce  Tp 

T43B 

30.65 

Fill  2 

Fit  Ops 

0 

49.90 

Fill  4 

FARP 

52S 

46.50 

Fill  4 

CAS  (on  call) 

STING 

HQII  /A55. 525 

No 

IFF: 


IFF  Mode 

Time 

Code 

Mode  1: 

0900-0930 

13 

0930-1000 

41 

1000-1030 

23 

1030-1100 

61 

1100-1130 

70 

1130-1200 

21 

Mode  2: 

N/A 

1721 

Mode  3: 

1200 

Mode  4: 

0000z-1200z 

A 

1200z-2400z 

B 
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Experimenter  Inputs: 

The  sortie  will  commence  with  the  Griffon  safely  airborne  at  the  2nd  pre-planned  Co-ordination  Point,  at 
an  altitude  of  400  feet  AGL,  on  track  and  heading  to  the  next  Co-ordination  Point.  Once  the  aircrew  is 
comfortable  the  simulation  will  be  uncaged;  the  time  will  be  0945Z. 

Experimenter  Inputs  (NOT  FOR  EXPERT  USER  CONSUMPTIONS 

These  activities  are  NOT  briefed  to  the  crew  -  in  order  to  allow  the  aircrew  to  interact  with  the  dynamic  of 
the  synthetic  environment  and  react  accordingly. 

The  following  dynamic  events  will  be  programmed  to  occur  during  the  “sortie”: 

.  5  x  Ml  045  HHMWV  TOWs  will  transit  NW  on  the  black  track  from  FL  983614  to  FL  968640.  The 

TOWs  need  to  be  moving  as  the  helicopter  is  transiting  NW  KNOWLTON  HILL. 

•  A  burning/crashed  SU-25  Frogfoot  will  be  on  the  ground  at  FL  930697. 

•  4  x  ZSU-234s  will  transit  South  on  the  road  from  FL  965765  to  FL  963746.  The  ZSU-234s  need  to 
be  moving  as  the  Griffon  reaches  BROAD  ROAD  CLEARING. 
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ANNEX  B 

Mission  Assessment  Questionnaires 
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TAMSS  Experiment  3 

Mission  Assessment 
Flying  Pilot  Version 


Participant  ID 

Date 

Time 

Mission  number 

Scenario  Code 

Condition 

MS  present  Or  MS  not  present 

Instructions 

If  you  have  additional  comments  about  any  question  or  item,  please  write  these  on 
the  back  of  the  corresponding  page. 
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Ml.  Rate  how  well  you  performed  of  each  of  the  following  tasks  during  THIS  mission. 
Circle  N/A  if  the  item  was  not  applicable  to  this  mission. 


PERFORMANCE 

Task 

very  very 

poor  adequate  good 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Cross  checking  relevant 

instruments/ symbology 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  aircraft  in 

recce  area 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  tactical 

flight 

1 

2 

3 

4 

5 

6 

7 

NA 
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M2.  Rate  the  difficulty  of  each  of  the  following  tasks  during  THIS  mission.  Circle  N/A  if 
the  item  was  not  applicable  to  this  mission. 


DIFFICULTY 

Task 

very  very 

easy  moderate  difficult 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  heading 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  altitude 

1 

2 

3 

4 

5 

6 

7 

NA 

Control  airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Cross  checking  relevant 

instruments/symbology 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  aircraft  in 

recce  area 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining  tactical  flight 

1 

2 

3 

4 

5 

6 

7 

NA 
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M3.  Rate  your  awareness  of  the  status  of  the  aircraft  systems  as  it  applies  to  THIS 
mission/task. 


AWARENESS 

very  moderate  very 

low  high 

Aircraft  Systems 

overall 

1 

2 

3 

4 

5 

6 

7 

NA 

Heading 

1 

2 

3 

4 

5 

6 

7 

NA 

RAD  Alt 

1 

2 

3 

4 

5 

6 

7 

NA 

Airspeed 

1 

2 

3 

4 

5 

6 

7 

NA 

Attitude 

1 

2 

3 

4 

5 

6 

7 

NA 
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M4.  Rate  your  awareness  of  tactical  information  relevant  to  THIS  mission  (i.e.  where  do 
you  need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the  mission 
unfolding). 


AWARENESS 

very  moderate  very 

low  high 

Overview  of  mission 

1 

2 

3 

4 

5 

6 

7 

NA 

Unfolding  of 

mission/keeping  track 

of  how  mission 

unfolds 

1 

2 

3 

4 

5 

6 

7 

NA 

Potential 

developments 

(anticipating  future 

scenarios) 

1 

2 

3 

4 

5 

6 

7 

NA 

Global  mission  goals 

1 

2 

3 

4 

5 

6 

7 

NA 

Specific  mission  goals 

1 

2 

3 

4 

5 

6 

7 

NA 

Enemy  activities 

1 

2 

3 

4 

5 

6 

7 

NA 

Friendly  activities 

1 

2 

3 

4 

5 

6 

7 

NA 

General  threat 

1 

2 

3 

4 

5 

6 

7 

NA 

Where  I  need  to  go 

1 

2 

3 

4 

5 

6 

7 

NA 
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M5.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location  of 
ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects  and 
landmarks  in  the  environment)  during  THIS  mission. 


AWARENESS 

very  moderate  very 

low  high 

Overall  Spatial 

Orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  target  (e.g. 

bridge) 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  enemy 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  friendly 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Target  location 

relative  to  enemy  and 

friendly  units 

1 

2 

3 

4 

5 

6 

7 

NA 

Important  landmarks 

1 

2 

3 

4 

5 

6 

7 

NA 

General  layout  of  the 

navigated  area 

1 

2 

3 

4 

5 

6 

7 

NA 
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M6.  Rate  the  following  crew  activity  during  THIS  mission. 


very  moderate  very 

low  high 

Overall  quality  of 

communication 

1 

2 

3 

4 

5 

6 

7 

NA 

The  usefulness  of 

information  provided 

by  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

NA 

The  usefulness  of 

information  provided 

by  the  ERSTA 

operator 

1 

2 

3 

4 

5 

6 

7 

NA 

Ability  to  convey 

information  to 

Mission  Commander 

1 

2 

3 

4 

5 

6 

7 

NA 
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M7.  Rate  your  workload  during  THIS  mission.  Circle  NA  if  not  applicable 
during  this  mission. 


WORKLOAD 

very  moderate  very 

low  high 

Planning  the  route? 

1 

2 

3 

4 

5 

6 

7 

NA 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  aircraft 

in  the  recce  zone 

1 

2 

3 

4 

5 

6 

7 

NA 

Radio  communications 

1 

2 

3 

4 

5 

6 

7 

NA 

Monitoring  friendly 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Monitoring  enemy 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Monitoring  general 

threats 

1 

2 

3 

4 

5 

6 

7 

NA 

Ingress  to  first 

waypoint 

1 

2 

3 

4 

5 

6 

7 

NA 

First  waypoint  to 

release  point  (RP) 

1 

2 

3 

4 

5 

6 

7 

NA 

Recce  zone  [release 

point  to  target] 

1 

2 

3 

4 

5 

6 

7 

NA 

Egress  [target  to  end] 

1 

2 

3 

4 

5 

6 

7 

NA 

Overall 

1 

2 

3 

4 

5 

6 

7 

NA 

Communication  with 

Mission  Commander 

1 

2 

3 

4 

5 

6 

7 

NA 

At  any  point,  did  you  find  that  your  workload  was  very  different  (e.g.,  much  higher  or  lower) 
than  across  the  mission  as  a  whole?  YES  or  NO 
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If  yes,  please  elaborate: 


If  at  any  point  you  found  that  your  workload  was  very  different,  what  aspects  of  your  tasks  (e.g., 
radio  communication,  using  infonnation  to  control  aircraft,  flying  aircraft,  etc.)  were  affected? 
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TLX  WORKLOAD  ASSESSMENT 

Instructions.  Place  an  X  on  each  scale  at  the  point  that  represents  the  magnitude  for  each  factor 
in  the  mission  you  just  performed.  Refer  to  the  Workload  Scale  Descriptions  for  definitions  of 
each  factor. 

SEGMENT  OF  MISSION:  Ingress  to  release  point 


Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  I — I — I — I — I — I — I — I — I — I — I  GOOD 


SEGMENT  OF  MISSION:  Release  point  to  first  observation  point 


Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  I — I — I — I — I — I — I — I — I — I — I  GOOD 
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SEGMENT  OF  MISSION:  Getting  and  maintaining  observation  of  target 

Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  | — | — | — | — | — | — | — | — | — | — |  GOOD 

SEGMENT  OF  MISSION:  Overall  activity  while  in  the  Recce  zone 
Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  I — I — I — I — I — I — I — I — I — I — I  GOOD 
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SEGMENT  OF  MISSION:  Egress 


Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  I — I — I — I — I — I — I — I — I — I — I  GOOD 
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TAMSS  Experiment  3 
Mission  Assessment 
Mission  Commander  Version 


Participant  ID 

Date 

Time 

Mission  number 

Scenario  Code 

Condition 

MS  present  Or  MS  not 

present 

Instructions 

If  you  have  additional  comments  about  any  question  or  item,  please  write  these  on 
the  back  of  the  corresponding  page. 
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Ml.  Rate  how  well  you  performed  on  each  of  the  following  tasks  during  THIS  mission. 
Circle  N/A  if  the  item  was  not  applicable  to  this  mission. 


PERFORMANCE 

Task 

very  very 

poor  adequate  good 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Navigate 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining/following  the  route 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  the  CDU 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  comms 

1 

2 

3 

4 

5 

6 

7 

NA 

Use  of  sensor  capability 

1 

2 

3 

4 

5 

6 

7 

NA 

Use  of  digital  map  capabilities 

1 

2 

3 

4 

5 

6 

7 

NA 

Communicating  with  the 

Mission  specialist 

1 

2 

3 

4 

5 

6 

7 

NA 

Guiding  the  Mission 

specialist  to  position  the 

sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

Guiding  the  flying  pilot  to 

position  the  aircraft  in 

recce  zone 

1 

2 

3 

4 

5 

6 

7 

NA 
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M2.  Rate  the  difficulty  of  each  of  the  following  tasks  during  THIS  mission.  Circle  N/A  if 
the  item  was  not  applicable  to  this  mission. 


DIFFICULTY 

Task 

very  very 

easy  moderate  difficult 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Navigate 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining/following  the  route 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  the  CDU 

1 

2 

3 

4 

5 

6 

7 

NA 

Using  comms 

1 

2 

3 

4 

5 

6 

7 

NA 

Use  of  sensor  capability 

1 

2 

3 

4 

5 

6 

7 

NA 

Use  of  tactical  map  capabilities 

1 

2 

3 

4 

5 

6 

7 

NA 

Communicating  with  the 

Mission  specialist 

1 

2 

3 

4 

5 

6 

7 

NA 

Guiding  the  Mission 

specialist  to  position  the 

sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

Guiding  the  flying  pilot  to 

position  the  aircraft  in 

recce  zone 

1 

2 

3 

4 

5 

6 

7 

NA 
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M3.  Rate  the  difficulty  of  getting  information  from  each  of  the  following  sources  during 
THIS  mission.  Circle  NA  if  not  applicable  to  this  mission. 


DIFFICULTY 

very  very 

easy  moderate  difficult 

The  ERST  A  sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

The  digital  tactical  map 

1 

2 

3 

4 

5 

6 

7 

NA 

Mission  Specialist 

1 

2 

3 

4 

5 

6 

7 

NA 

CDU 

1 

2 

3 

4 

5 

6 

7 

NA 
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M4.  Rate  your  awareness  of  tactical  information  relevant  to  your  mission  (i.e.  where  do 
you  need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the  mission 
unfolding)  during  THIS  mission. 


AWARENESS 

very  low  moderate  very  high 

Overview  of  mission 

1 

2 

3 

4 

5 

6 

7 

NA 

Keeping  track  of  how 

mission  unfolds 

1 

2 

3 

4 

5 

6 

7 

NA 

Potential  developments 

(anticipating  future 

events) 

1 

2 

3 

4 

5 

6 

7 

NA 

Global  mission  goals 

1 

2 

3 

4 

5 

6 

7 

NA 

Specific  mission  goals 

1 

2 

3 

4 

5 

6 

7 

NA 

Enemy  activities 

1 

2 

3 

4 

5 

6 

7 

NA 

Friendly  activities 

1 

2 

3 

4 

5 

6 

7 

NA 

General  threats 

1 

2 

3 

4 

5 

6 

7 

NA 

Where  I  need  to  go 

1 

2 

3 

4 

5 

6 

7 

NA 

80 


TAMSS  SA  Experiment  3 


M5.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location  of 
ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects  and 
landmarks  in  the  environment)  during  THIS  mission. 


AWARENESS 

very  low  moderate  very  high 

Overall  Spatial 

Orientation 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  objectives 

(e.g.  bridge) 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  enemy 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Ownship  location  in 

relation  to  friendly 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Target  location  relative 

to  enemy  and  friendly 

units 

1 

2 

3 

4 

5 

6 

7 

NA 

Important  landmarks 

1 

2 

3 

4 

5 

6 

7 

NA 

General  layout  of  the 

navigated  area 

1 

2 

3 

4 

5 

6 

7 

NA 

Information  provided  by 

the  ERST  A  system 

1 

2 

3 

4 

5 

6 

7 

NA 
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M6.  Rate  the  following  crew  activity  during  THIS  mission. 


very  low  moderate  very  high 

Overall  quality  of 

communication 

1 

2 

3 

4 

5 

6 

7 

NA 

The  usefulness  of 

information  provided  by 

the  Mission  specialist 

1 

2 

3 

4 

5 

6 

7 

NA 

Ability  to  instruct  the 

Mission  specialist 

1 

2 

3 

4 

5 

6 

7 

NA 

Awareness  of  Mission 

specialist  activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Ability  to  convey 

information  to  flying 

pilot 

1 

2 

3 

4 

5 

6 

7 

NA 

Coordinating  all  tasks 

1 

2 

3 

4 

5 

6 

7 

NA 

Coordinating 

communication 

1 

2 

3 

4 

5 

6 

7 

NA 
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M7.  Rate  your  workload  on  each  of  the  tasks  performed  during  THIS  mission 


WORKLOAD 

very  low  moderate  very  high 

Communicating  with 

flying  pilot 

1 

2 

3 

4 

5 

6 

7 

NA 

Communicating  with 

the  Mission  specialist 

1 

2 

3 

4 

5 

6 

7 

NA 

Navigating 

1 

2 

3 

4 

5 

6 

7 

NA 

Maintaining/ following 

the  route 

1 

2 

3 

4 

5 

6 

7 

NA 

Finding  waypoints 

1 

2 

3 

4 

5 

6 

7 

NA 

Guiding  the  flying  pilot  to 
position  the  aircraft  in  the 
recce  zone 

1 

2 

3 

4 

5 

6 

7 

NA 

Guiding  the  Mission 
specialist  to  position  the 
sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

Positioning  the  sensor 

1 

2 

3 

4 

5 

6 

7 

NA 

Operating  CDU 

1 

2 

3 

4 

5 

6 

7 

NA 

Radio  communications 

1 

2 

3 

4 

5 

6 

7 

NA 

Monitoring  friendly 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Monitoring  enemy 

activity 

1 

2 

3 

4 

5 

6 

7 

NA 

Monitoring  general 

threats 

1 

2 

3 

4 

5 

6 

7 

NA 

Ingress  to  first  waypoint 

1 

2 

3 

4 

5 

6 

7 

NA 

First  waypoint  to  release 

point  (RP) 

1 

2 

3 

4 

5 

6 

7 

NA 
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Recce  zone  [release 

point  to  target] 

1 

2 

3 

4 

5 

6 

7 

NA 

Egress  [target  to  end] 

1 

2 

3 

4 

5 

6 

7 

NA 

Overall 

1 

2 

3 

4 

5 

6 

7 

NA 
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TLX  WORKLOAD  ASSESSMENT 

Instructions.  Place  an  X  on  each  scale  at  the  point  that  represents  the  magnitude  of  each  factor  in 
the  mission  you  just  performed.  Refer  to  the  Workload  Scale  Descriptions  for  definitions  of  each 
factor. 

SEGMENT  OF  MISSION:  Ingress  to  release  point 


Mental  demand 

LOW  | — | — | — | — | — 

1—_ |  high 

Physical  demand 

LOW  | — | — | — | — | — 

— | — | — | — | — |  HIGH 

Temporal  demand 

LOW  | — | — | — | — | — 

HIGH 

Frustration  level 

LOW  | — | — | — | — | — 

— | — I — I — I — I  HIGH 

Effort 

LOW  | — | — |  -| — | — 

high 

Performance 

POOR  | — | — | — | — | — 

— 1 — I — I — I — I  GOOD 

SEGMENT  OF  MISSION:  Release  point  to  first  observation  point 


Mental  demand 

LOW  | — | — | — | — | — 

— | — | — | — | — |  HIGH 

Physical  demand 

LOW  | — | — | — | — | — 

— I — | — | — | — |  HIGH 

Temporal  demand 

LOW  | — | — | — | — | — 

— | — I — I — I — I  HIGH 

Frustration  level 

LOW  | — | — | — | — | — 

| — | — | — | — |  HIGH 

Effort 

LOW  | — | — | — | — | — 

high 

Perfonnance 

POOR  | — | — | — | — | — 

— | — | — | — | — |  GOOD 
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SEGMENT  OF  MISSION:  Getting  and  maintaining  observation  of  target 

Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  | — | — | — | — | — | — | — | — | — | — |  GOOD 

SEGMENT  OF  MISSION:  Overall  activity  while  in  Recce  zone. 

Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  | — | — | — | — | — | — | — | — | — | — |  GOOD 

SEGMENT  OF  MISSION:  Egress 

Mental  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Physical  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Temporal  demand  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Frustration  level  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 
Effort  LOW  | — | — | — | — | — | — | — | — | — | — |  HIGH 

Performance  POOR  I — I — I — I — I — I — I — I — I — I — I  GOOD 
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Open-ended  questions 

When  there  was  no  Mission  Specialist  present,  would  you  see  value  in  having  a  Mission 
specialist? 

YES  or  NO  Please  elaborate: 


When  there  was  a  Mission  Specialist  present  would  you  see  value  in  not  having  a  Mission 
specialist?  YES  or  No  Please  elaborate: 


Would  it  be  beneficial  to  have  a  Mission  Commander  override  for  the  ERSTA  system?  YES  or 
NO 

Please  elaborate: 


87 


TAMSS  SA  Experiment  3 


Was  there  a  time  when  you  needed  to,  but  could  not  operate  the  sensor  (e.g.  the  Mission 
specialist  had  locked  the  system)  YES  or  NO 
Please  elaborate: 


At  any  point,  did  you  find  that  your  workload  was  very  different  (e.g.,  much  higher  or  lower) 
than  across  the  mission  as  a  whole?  YES  or  NO 
If  yes,  please  elaborate: 
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If  at  any  point  you  found  that  your  workload  was  very  different,  what  aspects  of  your  tasks  (e.g., 
radio  communication,  using  infonnation  to  control  aircraft,  flying  aircraft,  etc.)  were  affected? 
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ANNEX  C 

Background  Questionnaire  for  Pilots 
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TAMSS  Experiment  3  Participant  ID: _ 

Background  Information:  Pilots 

This  information  helps  us  to  determine  whether  we  have  recruited  a  broad  sample  of  pilots.  To 
get  an  unbiased  understanding  of  the  concerns  of  the  operation  community,  we  must  get 
feedback  from  a  cross-section  of  experience  levels  and  backgrounds. 

1 .  Age: _  (in  years) 

2.  Years  in  CF: _ 

3.  Years  as  a  pilot: _ 

4.  TacHel  experience: _ (in  years) 

5.  Rank: _ 

6.  Handedness:  _ left  OR _ right 


Estimated  Flight  Time 


Type  of  Flight 

Estimated  time  (in  hours) 

1.  Total  flight  time 

2.  Total  rotary  wing 

3.  Total  Griffon 

4.  Time  spent  using  a  flight  simulator 

Specify  the  type  of  simulator  experiences  that  you  have  had: 


Indicate  any  other  experience  (e.g.  TacHel)  that  you  believe  may  be  relevant  to  your 
performance  in  this  experiment. 
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ANNEX  D 

Background  Questionnaire  for  Mission  Specialist 
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TAMSS  Experiment  3  Participant  ID: _ 

Background  Information:  Mission  Specialist 

This  information  helps  us  to  determine  whether  we  have  recruited  a  broad  sample  of  pilots.  To 
get  an  unbiased  understanding  of  the  concerns  of  the  operation  community,  we  must  get 
feedback  from  a  cross-section  of  experience  levels  and  backgrounds. 

7.  Age: _  (in  years) 

8.  Years  in  CF: _ 

9.  Years  as  a  Mission  specialist: _ 

10.  TacHel  experience: _  (in  years) 

11.  MOC  (Millitary  Occupation  Code): _ 

12.  Rank: _ 

13.  Handedness:  _ left  OR _ right 


Estimated  Flight  Time 


Type  of  Flight 

Estimated  time  (in  hours) 

2.  Total  rotary  wing 

3.  Total  Griffon 

4.  Time  spent  using  a  flight  simulator 

Specify  the  type  of  simulator  experiences  that  you  have  had: 


Indicate  any  other  experience  (e.g.  TacHel)  that  you  believe  may  be  relevant  to  your 
performance  in  this  experiment. 
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ANNEX  E 


Summary  Data:  Mission  Assessment  Questionnaires 
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Summary  of  Post  Mission  Questionnaires 
TAMSS  Project  -  Experiment  2 
Flying  Pilot 

Rated  Performance 

Ml.  Rate  how  well  you  performed  of  each  of  the  following  tasks  during  THIS  mission.  Circle 
N/A  if  the  item  was  not  applicable  to  this  mission. 

Average  scores;  poor  (1-3),  adequate  (4)  and  good  (5-7) 

Average  rated  performance  overall;  MS  present=5.107,  MS  not  present=4.732 


MS  present 

MS  not  present 

Finding  waypoints 

Adequate  (4.375) 

Adequate  (4.625) 

Control  heading 

Good  (5.25) 

Good  (5) 

Control  altitude 

Good  (5.25) 

Adequate  (4.625) 

Control  airspeed 

Good  (5.625) 

Adequate  (4.625) 

Cross  checking  relevant  instrument/symbology 

Adequate  (4.875) 

Adequate  (4.875) 

Positioning  the  aircraft  in  recce  area 

Good  (5.375) 

Adequate  (4.625) 

Maintaining  tactical  flight 

Good  (5) 

Adequate  (4.75) 
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Rated  Task  Difficulty 

M2.  Rate  the  difficulty  of  each  of  the  following  tasks  during  THIS  mission.  Circle  N/A  if 
the  item  was  not  applicable  to  this  mission. 

Average  scores;  easy  (1-3),  moderate  (4)  and  difficult  (5-7) 

Average  rated  task  difficulty  overall;  MS  present=3.21,  MS  not  present=3.23 


MS  present 

MS  not  present 

Finding  waypoints 

Easy  (2.75) 

Easy  (2.625) 

Control  heading 

Easy  (3.125) 

Easy  (3.125) 

Control  altitude 

Easy  (3.625) 

Easy  (3.375) 

Control  airspeed 

Easy  (2.75) 

Easy  (3) 

Cross  checking  relevant  instrument/symbology 

Easy  (3.125) 

Easy  (3.375) 

Positioning  the  aircraft  in  recce  area 

Easy  (3.375) 

Easy  (3.625) 

Maintaining  tactical  flight 

Easy  (3.75) 

Easy  (3.5) 
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Rated  Situation  Awareness 

M3.  Rate  your  awareness  of  the  status  of  the  aircraft  systems  as  it  applies  to  THIS 
mission/task. 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  situation  awareness  overall;  MS  present=5.625, 

MS  not  present=  5.12 


MS  present 

MS  not  present 

Aircraft  systems 
overall 

High 

(5.25) 

Moderate 

(4.625) 

Heading 

High  (5.5) 

High  (5) 

RAD  alt 

High 

(6.125) 

High  (5.75) 

Airspeed 

High 

(5.375) 

High  (5) 

Attitude 

High 

(5.875) 

High  (5.25) 
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Rated  tactical  awareness 

M4.  Rate  your  awareness  of  tactical  infonnation  relevant  to  THIS  mission  (i.e.  where  do 
you  need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the 
mission  unfolding). 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  tactical  awareness  overall;  MS  present=5.125,  MS  not  present=5.166 


MS  present 

MS  not  present 

Overview  of  mission 

High  (5.875) 

High  (5.375) 

Unfolding  of  mission/keeping  track  of  how 
mission  unfolds 

Moderate  (5) 

High  (5) 

Potential  developments  (anticipating  future 
scenarios) 

High  (6) 

Low  (4.875) 

Global  mission  goals 

High  (5.25) 

High  (5.125) 

Specific  mission  goals 

High  (4.875) 

High  (5.125) 

Enemy  activities 

Low  (4.375) 

High  (4.375) 

Friendly  activities 

Low  (4.625) 

Low  (3.625) 

General  threat 

Moderate  (5.125) 

Moderate  (4) 

Where  I  need  to  go 

Low  (5) 

High  (4.5) 
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Rated  spatial  awareness 

M5.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location  of 
ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects  and 
landmarks  in  the  environment)  during  THIS  mission. 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  spatial  awareness  overall;  MS  present=4,  MS  not  present=4.3928 


MS  present 

MS  not  present 

Overall  spatial  orientation 

Moderate 

(4.875) 

High  (5) 

Ownship  location  in  relation  to  target  (e.g. 
bridge) 

High  (5.25) 

Moderate 

(4.625) 

Ownship  location  in  relation  to  enemy  activity 

Moderate  (4) 

Moderate 

(4.375) 

Ownship  location  in  relation  to  friendly  activity 

NA 

Low  (3.625) 

Target  location  relative  to  enemy  and  friendly 
units 

Low  (3.75) 

Low  (3.625) 

Important  landmarks 

Moderate  (4.75) 

Moderate 

(4.625) 

General  layout  of  the  navigated  area 

Low  (5.375) 

High  (4.875) 
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Rated  Crew  activity 

M6.  Rate  the  following  crew  activity  during  THIS  mission. 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  crew  activity  overall;  MS  present=4.718,  MS  not  present=3.84 


MS  present 

MS  not  present 

Overall  quality  of 
communication 

Moderate  (4.875) 

High  (4.5) 

The  usefulness  of 
information  provided  by 
Mission  Commander 

Moderate  (4.75) 

Moderate  (4.625) 

The  usefulness  of 

information  provided  by  the 

ERSTA  operator 

Moderate  (4.875) 

Low  (1.75) 

Ability  to  convey 

Moderate  (4.375) 

Moderate  (4.5) 

information  to  Mission 

Commander 
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Rated  Workload 

M7.  Rate  your  workload  during  THIS  mission.  Circle  NA  if  not  applicable  for  this 
mission. 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  workload  overall;  MS  present=  2.673,  MS  not  present=3.1 15 


MS  Present 

MS  not  present 

Planning  the  route? 

Low  (2.5) 

Low  (3.125) 

Finding  waypoints 

Low  (2.75) 

Low  (3.375) 

Positioning  the 

aircraft  in  the  recce 

zone 

Low  (3.5) 

Moderate  (4) 

Radio 

communications 

Low  (.875) 

Low  (1.25) 

Monitoring  friendly 

activity 

Low  (1.625) 

Low  (1.875) 

Monitoring  enemy 

activity 

Low  (2.875) 

Moderate  (4) 

Monitoring  general 

threats 

Low  (3.875) 

Moderate  (4) 

Ingress  to  first 

waypoint 

Low  (1.875) 

Low  (2.125) 

First  waypoint  to 

release  point  (RP) 

Low  (2) 

Low  (2.25) 

Recce  zone  [release 

point  to  target] 

Low  (3.5) 

Moderate  (4) 

Egress  [target  to  end] 

Low  (2.375) 

Low  (3) 

Overall 

Low  (3.375) 

Low  (3.875) 

Communication  with 

Mission  Commander 

Low  (3.625) 

Low  (3.625) 
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Additional  questions: 

1 .  At  any  point,  did  you  find  that  your  workload  was  very  different  (e.g.,  much  higher  or 
lower)  than  across  the  mission  as  a  whole?  YES  or  NO 

If  yes,  please  elaborate: 

P2:  (MS  not  present)  No.  I  fly  where  the  Mission  Commander  tells  me  to  fly. 

P2:  (MS  present)  No. 

P3:  (MS  present)  Yes.  Moving  into  first  and  subsequent  observation  positions  and 
keeping  helicopter  control  while  in  NOE  and  masked  locations. 

P3:  (MS  not  present)  Yes.  Need  to  provide  input  verbally  to  MC  regarding  my  intent  (i.e., 
where  to  best  tactically  position  the  aircraft).  Also,  proving  suggestions  of  future 
intentions  and  possible  best  courses  of  action. 

P4:  (MS  not  present)  Yes.  Workload  was  higher  when  trying  to  get  set  up  in  the 
observation  points. 

P4:  (MS  present)  Yes.  A  definite  peak  of  workload  just  maintaining  hover  at  observation 
point  before  the  objective  and  also  some  work  to  ensure  accurate  position  at  other 
observation  points. 

P5:  (MS  present)  No. 

P5:  (MS  not  present)  Yes,  when  engaged  by  enemy  fire. 

P6:  (MS  present)  Yes.  During  enemy  engagements. 

P6:  (MS  not  present)  No. 

P7:  (MS  present)  Yes.  When  flying  across  the  “swamp”  there  were  no  references  to 
indicate  altitude,  attitude,  or  groundspeed. 

P7  (MS  not  present)  Yes.  In  the  hover,  at  times  it  was  difficult  to  maintain  height  and 
position  due  to  lack  of  references  and  sensitivity  of  flight  controls. 

2.  If  at  any  point  you  found  that  your  workload  was  very  different,  what  aspects  of  your 
tasks  (e.g.,  radio  communication,  using  infonnation  to  control  aircraft,  flying  aircraft, 
etc.)  were  affected? 

P2:  (MS  not  present)  Sometimes  there  were  lack  of  directions  from  the  Mission 
Commander. 

P3:  (MS  present)  Fairly  high  workload  to  keep  aircraft  in  correct  position  to  allow  sensor 
operator  to  view  the  target  area. 

P3  (MS  not  present)  As  above.  Since  MC  was  too  heavily  task-saturated  to  control 
mission,  direct  aircraft,  or  make  radio  calls.  I  was  able  to  provide  a  wider  SA  of  the 
situation  while  MC  was  focused  on  tasks.  I  had  more  autonomy  to  position  aircraft  into 
the  best  position. 

P4:  (MS  not  present)  Flying  the  aircraft  became  a  little  more  challenging. 

P5:  (MS  not  present)  Flying  the  aircraft;  navigation. 

P6:  (MS  present)  Controlling  the  aircraft  to  aid  the  ERSTA  operator. 

P7:  (MS  present)  Flying  aircraft  across  the  “marsh”  was  very  difficult  due  to  lack  of 
references. 

P7  (MS  not  present)  As  indicated  above.  Control  of  aircraft  in  hover  was  difficult  at 
times. 
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P8:  (MS  present)  There  was  a  lot  of  communication  going  on  between  MC  and  MS  and  it 
kept  me  from  asking  and  getting  information  about  where  the  enemy  is  in  relation  to 
aircraft. 
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TLX  WORKLOAD  ASSESSMENT 

Instructions.  Place  an  X  on  each  scale  at  the  point  that  represents  the  magnitude  for  each 
factor  in  the  mission  you  just  performed.  Refer  to  the  Workload  Scale  Descriptions  for 
definitions  of  each  factor. 

Average  scores;  low  (1-4),  moderate  (5)  and  high  (6-10) 


SEGMENT  OF  MISSION:  Ingress  to  release  point 


MS  present 

MS  not  present 

Mental  demand 

2.0625 

2.3125 

Physical  demand 

2.1875 

2.1875 

Temporal  demand 

1.9375 

2.0000 

Frustration  level 

2.0625 

1.8750 

Effort 

2.3750 

2.0625 

Performance 

8.4375 

8.3125 

SEGMENT  OF  MISSION:  Release  point  to  first  observation  point 


MS  present 

MS  not  present 

Mental  demand 

3.7500 

3.5625 

Physical  demand 

3.0000 

3.5625 

Temporal  demand 

3.0625 

3.5000 

Frustration  level 

1.9375 

3.2500 

Effort 

3.6250 

3.9375 

Performance 

7.9375 

7.6875 
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SEGMENT  OF  MISSION:  Getting  and  maintaining  observation  of  target 


MS  present 

MS  not  present 

Mental  demand 

4.1875 

5.0625 

Physical  demand 

3.8750 

4.3750 

Temporal  demand 

3.2500 

5.1250 

Frustration  level 

3.0625 

4.6875 

Effort 

4.9375 

5.3750 

Performance 

8.5000 

6.6875 

SEGMENT  OF  MISSION:  Overall  activity  while  in  the  Recce  zone 


MS  present 

MS  not  present 

Mental  demand 

4.8125 

4.5000 

Physical  demand 

4.0000 

4.5000 

Temporal  demand 

4.1250 

4.3125 

Frustration  level 

3.0000 

4.0625 

Effort 

4.8750 

5.6250 

Performance 

8.4375 

7.1875 
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SEGMENT  OF  MISSION:  Egress 


MS  present 

MS  not  present 

Mental  demand 

2.6250 

3.2500 

Physical  demand 

2.5000 

2.2500 

Temporal  demand 

3.1875 

2.7500 

Frustration  level 

1.3750 

2.4375 

Effort 

2.6875 

3.1250 

Performance 

8.5625 

8.5000 
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Summary  of  Post  Mission  Questionnaires  TAMSS  Project  -  Experiment  3 
Mission  Commander  Rated  Confidence  in  Re-creating  Mission 


Rated  performance 

M 1 .  Rate  how  well  you  performed  on  each  of  the  following  tasks  during  THIS 

mission.  Circle  N/A  if  the  item  was  not  applicable  to  this  mission. 

Average  scores;  poor  (1-3),  adequate  (4)  and  good  (5-7) 

Average  rated  perfonnance  overall;  MS  present=4.238,  MS  not  present=2.875 


MS  present 

MS  not  present 

Finding  waypoints 

Good  (5.5) 

Adequate  (4.5) 

Navigate 

Good  (5.625) 

Adequate(4.625) 

Maintaining/following  the  route 

Good  (5) 

Adequate  (4.5) 

Using  the  CDU 

Using  comms 

Poor  (3.125) 

Poor  (3.75) 

Use  of  sensor  capability 

Adequate  (4) 

Adequate  (3.125) 

Use  of  digital  map  capabilities 

Good  (5.25) 

Poor  (3.875) 

Communicating  with  the  Mission 

specialist 

Good  (5.375) 

NA 

Guiding  the  Mission 

specialist  to  position  the 

sensor 

Good  (5.75) 

.NA 

Positioning  the  sensor 

Poor  (1.5) 

Poor  (3) 

Guiding  the  flying  pilot  to 

position  the  aircraft  in  recce 

zone 

Good  (5.5) 

Adequate  (4.25) 
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Rated  Task  Difficulty 

M2.  Rate  the  difficulty  of  each  of  the  following  tasks  during  THIS  mission.  Circle  N/A  if 
the  item  was  not  applicable  to  this  mission. 

Average  scores;  easy  (1-3),  moderate  (4)  and  difficult  (5-7) 

Average  rated  task  difficulty  overall;  MS  present=2.383,  MS  not  present=2.863 


MS  present 

MS  not  present 

Finding  waypoints 

Easy  (3.125) 

Easy  (3.5) 

Navigate 

Easy  (2.875) 

Easy  (3.25) 

Maintaining/following  the  route 

Easy  (2.5) 

Easy  (3.125) 

Using  the  CDU 

NA 

NA 

Using  comms 

Easy  (1.875) 

Easy  (3.375) 

Use  of  sensor  capability 

Easy  (3) 

Difficult  (5.375) 

Use  of  tactical  map  capabilities 

Easy  (3.5) 

Easy  (3.625) 

Communicating  with  the 

Mission  specialist 

Easy  (2.625) 

NA 

Guiding  the  Mission 

specialist  to  position  the 

sensor 

Easy  (2.625) 

NA 

Positioning  the  sensor 

Easy  (1.375) 

Difficulty  (5.375) 

Guiding  the  flying  pilot  to 

position  the  aircraft  in 

recce  zone 

Easy  (2.75) 

Easy  (3.875) 

Rated  difficulty  getting  information 

M3.  Rate  the  difficulty  of  getting  information  from  each  of  the  following  sources  during 
THIS  mission.  Circle  NA  if  not  applicable  to  this  mission. 

Average  scores;  easy  (1-3),  moderate  (4)  and  difficult  (5-7) 

Average  rated  difficulty  getting  infonnation  overall;  MS  present-  2.214, 

MS  not  present=  2.09 


MS  present 

Ms  not  present 

The  ERSTA  sensor 

Easy  (3.5) 

Moderate  (4.875) 

The  digital  tactical  map 

Easy  (2.8571) 

Easy  (3.5) 

Mission  Specialist 

Easy  (2.5) 

NA 

CDU 

NA 

NA 
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Rated  Situation  Awareness 

M4.  Rate  your  awareness  of  tactical  information  relevant  to  your  mission  (i.e.  where  do 
you  need  to  go  and  what  needs  to  be  completed,  mission  goals  and  how  is  the 
mission  unfolding)  during  THIS  mission 
Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  situation  awareness  overall;  MS  present=  5.375,  MS  not  prcsent=  4.43 


MS  present 

MS  not 
present 

Overview  of  mission 

High  (6) 

High  (5.125) 

Unfolding  of  mission/keeping  track  of  how  mission 
unfolds 

High  (5.375) 

Moderate 

(4.625) 

Potential  developments  (anticipating  future 
scenarios) 

High  (5.375) 

Low  (3.5) 

Global  mission  goals 

High  (5.75) 

Moderate 

(4.625) 

Specific  mission  goals 

High  (5.5) 

High  (5.125) 

Enemy  activities 

High  (5.25) 

Low  (3.875) 

Friendly  activities 

High  (5) 

Moderate 

(4.25) 

General  threat 

Moderate 

(4.875) 

Low  (3.625) 

Where  I  need  to  go 

High  (5.25) 

High  (5.125) 

109 


TAMSS  SA  Experiment  3 


Rated  spatial  awareness 

M5.  Rate  your  spatial/navigational  awareness  (i.e.  where  you  need  to  go  and  location 
of  ownship  in  relation  to  enemies,  friendly  units,  target  and  other  relevant  objects 
and  landmarks  in  the  environment)  during  THIS  mission.  Average  scores;  Low 
(1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  spatial  awareness  overall;  MS  present  5.4,  MS  not  present=4.5 


MS  present 

MS  not  present 

Overall  Spatial 

Orientation 

High  (5.625) 

Moderate  (4.875) 

Ownship  location  in 

relation  to  objectives 

(e.g.  bridge) 

High  (5.5) 

Moderate  (4.75) 

Ownship  location  in 

relation  to  enemy 

activity 

High  (5.375) 

Moderate  (4.25) 

Ownship  location  in 

relation  to  friendly 

activity 

High  (5.125) 

Moderate  (4.375) 

Target  location 

relative  to  enemy  and 

friendly  units 

High  (5.25) 

Moderate  (4.125) 

Important  landmarks 

High  (5.5) 

Moderate  (4.875) 

General  layout  of  the 

navigated  area 

High  (5.75) 

Moderate  (4.875) 

Information  provided 

by  the  ERSTA 

system 

High  (5.125) 

Low  (3.875) 
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Rated  crew  activity 

M6.  Rate  the  following  crew  activity  during  THIS  mission. 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  crew  activity;  MS  present=5.32,  MS  not  present=2.446 


MS  present 

MS  not  pres 

Overall  quality  of 

communication 

High  (5.125) 

Moderate  (4.125) 

The  usefulness  of 

infonnation  provided 

by  the  Mission 

specialist 

High  (5.875) 

.0000 

Ability  to  instruct  the 

Mission  specialist 

High  (5.25) 

.0000 

Awareness  of  Mission 

specialist  activity 

High  (5.375) 

.0000 

Ability  to  convey 

infonnation  to  flying 

pilot 

High  (5.375) 

Moderate  (4.75) 

Coordinating  all  tasks 

High  (5.25) 

Moderate  (4) 

Coordinating 

communication 

5.0000 

4.2500 
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Rated  Workload 

M7.  Rate  your  workload  during  THIS  mission.  Circle  NA  if  not  applicable  for  this 
mission. 

Average  scores;  low  (1-3),  moderate  (4)  and  high  (5-7) 

Average  rated  workload;  MS  present=3.10,  MS  not  present=  3.35 


MS  Present 

MS  not  present 

Communicating  with  flying 

pilot 

Low  (3) 

Moderate  (4) 

Communicating  with  the 

Mission  specialist 

Low  (3.0833) 

.5000 

Navigating 

Moderate  (4) 

Moderate  (4.25) 

Maintaining/following  the 

route 

Low  (3.8333) 

Low  (3.5833) 

Finding  waypoints 

Low  (3.5) 

Low  (3.3333) 

Guiding  the  flying  pilot  to 
position  the  aircraft  in  the  recce 
zone 

Low  (3.9167) 

Moderate  (4.4167) 

Guiding  the  Mission 
specialist  to  position  the 
sensor 

Low  (2.75) 

.5000 

Positioning  the  sensor 

Low  (2) 

Low  (5.5000) 

Operating  CDU 

.0000 

.0000 

Radio  communications 

Low  (2.75) 

Moderate  (4.6364) 

Monitoring  friendly  activity 

Low  (2.75) 

Low  (3.3333) 

Monitoring  enemy  activity 

Moderate  (4.1667) 

Moderate  (4.75) 

Monitoring  general  threats 

Moderate  (4) 

Moderate  (4.4167) 

Ingress  to  first  waypoint 

Low  (2.6667) 

Low  (2.4167) 

First  waypoint  to  release 

point  (RP) 

Low  (2.75) 

Low  (2.75) 

Recce  zone  [release  point  to 

target] 

Moderate  (4.0833) 

Moderate  (4.2727) 

Egress  [target  to  end] 

Low  (2.75) 

Low  (3) 

Overall 

Low  (3.9167) 

Moderate  (4.75) 
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Open-ended  questions 

1 .  When  there  was  no  Mission  Specialist  present,  would  you  see  value  in  having  a  Mission 
specialist?  YES  or  NO.  Please  elaborate: 

PI:  (MS  not  present)  Yes.  Are  you  kidding?  First  of  all,  he  is  qualified  recce.  Second,  he  had 
experience  with  ERSTA.  Third,  he  has  control  I  do  not  have  up  front.  Finally,  it  would  reduce 
the  MC’s  workload. 

P2:  (MS  not  present)  Yes.  A  lot  less  workload  for  me.  MS  can  do  communications,  operate 
sensors  and  help  me  evaluate  best  course  of  action. 

P3:  (MS  not  present)  Yes.  Absolutely! !  MC  was  in  “overload”  mode  for  most  of  the  recce  area 
operations  trying  to  both  give  directions  to  the  FP  plus  operating  the  sensor.  Adding 
communications  with  outside  agencies  made  the  work  flow  even  busier.  MC  was  too  involved  in 
detailed  operation  of  sensor  to  achieve  good  overall  SA. 

P4:  (MS  not  present)  Yes.  Absolutely.  Without  the  MS,  I  felt  task  saturated — much  more  busy, 
the  quality  of  the  mission  (i.e.,  the  ability  to  gather  infonnation)  was  significantly  reduced. 
Frustration  levels  would  be  less  and  communication  would  be  enhanced  with  the  addition  of  a 
MS. 

P5:  (MS  not  present)  Yes.  Only  a  slight  increase  in  enemy  contact  would  have  overloaded  MC 
task  load. 

P6:  (MS  not  present)  Yes.  This  would  greatly  decrease  the  workload.  The  need  to  decrease  the 
workload  on  the  MC  was  required  in  this  mission. 

P7:  (MS  not  present)  Yes.  I  could  have  spent  more  time  focusing  on  the  big  picture  while  the  MS 
could  have  put  the  sensor  to  better  use. 

P8:  (MS  not  present)  Yes.  It  is  easier  to  supervise,  direct  and  follow  the  mission.  When  I  was 
absorbed  by  the  sensor  I  did  not  notice  the  aircraft  had  turned  and  we  were  exposing  the  side  of 
the  aircraft,  which  made  us  more  visible  to  the  enemy. 

2.  When  there  was  a  Mission  Specialist  present  would  you  see  value  in  not  having  a  Mission 
specialist?  YES  or  NO.  Please  elaborate: 

PI:  (MS  present)  No.  The  MS  is  much  more  aware  of  what  is  going  on.  He  has  key  personnel 
(qualified  crew)  using  them  to  his  advantage  and  thus  he  (MC)  has  much  more  time  to  control 
everything. 

P2:  (MS  present)  No.  It  is  very  hard  to  navigate  and  operate  ERSTA  at  the  same  time.  Especially 
if  you  have  a  second  a/c  to  control 
P3:  (MS  present)  No. 

P4:  (MS  present)  No.  The  MS’s  contribution  to  the  mission  was  instrumental  to  its  success. 

P4:  (MS  not  present)  No. 

P5:  (MS  present)  No.  My  workload  would  have  doubled. 

P6:  (MS  present)  No.  The  workload  reduced  by  having  an  MS.  Greatly  improved  the  ability  to 
complete  the  mission  within  the  time  frame  allocated. 

P7:  (MS  present)  No.  The  MS  was  able  to  conduct  many  of  the  tasks  while  MC  could  focus  on 
the  larger  picture. 
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P8:  (MS  present)  No.  It  decreases  the  workload  by  having  someone  taking  care  of  some  of  the 
radio  transmissions.  Using  the  sensor  would  keep  me  from  looking  outside  and  at  the  map. 
Having  a  MS  I  can  only  glance  at  the  image  without  having  to  adjust  it. 

P8:  (MS  not  present)  No. 


3.  Would  it  be  beneficial  to  have  a  Mission  Commander  override  for  the  ERSTA  system?  YES 
or  NO.  Please  elaborate: 

PI :  (MS  present)  No.  Unless  there  is  a  problem  with  the  system  (i.e.,  with  the  controls  in  the 
back). 

P2:  (MS  present)  No.  But  as  MC,  I  could  give  the  MS  the  word  of  command  to  let  me  have 
control  of  the  system  or  guide  him  on  the  target  I  want  to  see. 

P2:  (MS  not  present)  No.  But  as  MC,  I  could  give  the  MS  the  word  of  command  to  let  me  have 
control  of  the  system  or  guide  him  on  the  target  I  want  to  see. 

P3:  (MS  present)  Yes.  At  certain  times  this  may  be  necessary  but  for  the  majority  of  the  time  the 
MS  operated  the  senor  and  provided  the  necessary  input. 

P4:  (MS  present)  Yes.  If  the  MS  became  incapacitated  due  to  any  reason  then  the  MC  should  be 
able  to  use  the  system. 

P4:  (MS  not  present)  Yes.  If  the  mission  specialist  was  incapable  of  performing  the  required  task 
it  could  be  then  handed  over  to  the  MC. 

P5:  (MS  present)  Yes.  Only  when  required  for  short  duration  to  orient  sensor  operator  to  MC 
desired  area  of  observation. 

P6:  (MS  present)  Yes.  It  would  be  required  by  the  MC  to  have  the  ability  to  redirect  the  MS  by 
physically  moving  the  sensor.  The  MC  could  also  take  control  to  view  an  area  of  concern. 

P7:  (MS  present)  No.  It  was  easy  to  redirect  the  MS  onto  areas  of  interest. 

P8:  (MS  present)  Yes.  If  after  trying  to  explain  to  the  MS  where  to  look  with  the  sensor  without 
success,  then  having  an  override  could  cut  down  time  spent  trying  to  direct  someone  to  direct  the 
sensor.  Keep  in  mind  that  the  MS  frustration  level  could  go  up  slightly. 

P8:  (MS  not  present)  Yes. 

4.  Was  there  a  time  when  you  needed  to,  but  could  not  operate  the  sensor  (e.g.  the  Mission 
specialist  had  locked  the  system)  YES  or  NO  Please  elaborate: 

PI :  (MS  not  present)  Yes.  To  lock  the  camera  on  the  AC’s  heading  would  have  facilitated  my 
job.  It  is  not  realistic  to  operate  this  way  because  mission  accomplishment  would  have  very  low 
chance  of  completion. 

PI :  (MS  not  present)  Yes.  Trying  to  align  the  camera  toward  the  targets. 

P2:  (MS  present)  No. 

P3:  (MS  present)  No. 

P4:  (MS  present)  No.  At  no  point  did  I  feel  it  necessary  to  take  control  of  the  sensor.  The  MS 
was  doing  a  great  job  and  I  was  able  to  focus  on  other  aspects  of  the  mission. 

P4:  (MS  not  present)  No. 

P5:  (MS  present)  No. 

P6:  (MS  present)  No.  It  was  easy  to  communicate  between  the  MC  and  MS  to  determine  who 
was  in  control  of  the  asset. 
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P7:  (MS  present)  No.  I  did  not  feel  the  need  to  operate  the  sensor. 

P8:  (MS  present)  No.  I  did  not  need  to  use  it.  It  was  pointed  exactly  where  I  wanted  it. 

5.  At  any  point,  did  you  find  that  your  workload  was  very  different  (e.g.,  much  higher  or  lower) 
than  across  the  mission  as  a  whole?  YES  or  NO  If  yes,  please  elaborate: 

PI:  (MS  not  present)  Yes,  doing  contact  report,  calling  artillery  onto  enemy  positions  and  trying 
to  align  camera  while  zooming  in. 

PI:  (MS  present)  Yes.  Much  lower  workload  due  to  excellent  work  done  by  mission  specialist 
P2:  (MS  present)  Yes.  Giving  direction  to  the  pilot  when  in  or  close  to  an  observing  position. 

P2:  (MS  not  present)  Yes.  The  workload  is  a  lot  more  when  in  contact  with  the  enemy. 

P3:  (MS  present)  Yes.  During  the  first  NOE  ,  tactical  movement  from  release  point  to  first  area 
of  observation.  Some  of  this  was  due  to  limited  familiarity  with  both  the  aircraft  and  sensor 
systems.  Also,  first  time  working  as  a  crew  so  developing  out  CRM  and  interpersonal 
interactions. 

P3:  (MS  not  present)  Yes.  Workload  while  observing  target  area,  engaging  with  artillery  and 
give  SITREPs  was  intense. 

P4:  (MS  present)  Yes.  In  the  observation  point  when  we  had  eyes  on  the  objective  there  was 
more  radio  communication  and  the  helicopter  required  a  bit  more  manoeuvering  to  get  into 
position  thus  more  coordination  was  required  on  my  behalf. 

P4:  (MS  not  present)  Yes.  Within  the  recce  zone — the  workload  significantly  increased  (more 
communications,  more  manipulation  of  the  sensor,  more  manoeuvering  and  hence  navigation  to 
get  into  a  good  location  to  observe  the  object). 

P5:  (MS  present)  Yes.  When  the  FP  identified  artillery  impact  in  vicinity  of  OP3. 

P5:  (MS  not  present)  Yes.  Once  enemy  contact  was  achieved,  I  had  higher  workload 
(maintaining  contact  to  the  task  list,  navigating,  communicating,  etc.). 

P6:  (MS  present)  Yes.  The  requirement  to  reposition  the  helicopter  to  better  utilize  the  ERSTA 
required  an  increased  workload.  At  this  point,  the  MC  had  to  redirect  the  MS  search  area,  find  a 
new  observation  point  and  direct  the  flying  pilot  to  the  new  observation  point. 

P6:  (MS  not  present)  Yes.  At  the  recce  zone.  Just  too  many  things  to  do.  Any  interruption  or 
break  in  flow  would  overload  the  MC. 

P7:  (MS  present)  Yes.  In  the  recce  area  there  were  more  tasks  to  complete  and  maintain  SA. 

P7:  (MS  not  present)  Yes.  With  additional  sensor  duties,  from  release  point  to  egress  workload 
was  higher  causing  me  to  work  harder.  However,  overall  wit  this  higher  workload,  I  felt  I 
performed  better  than  in  the  run  with  a  MS  although  the  full  utilization  of  the  sensor’s 
capabilities  suffered. 

P8:  (MS  present)  Yes.  Slightly  higher  after  the  release  point  until  the  end  of  the  mission. 

P8:  (MS  not  present)  Yes.  Whenever  I  had  to  operate  the  sensor  to  find  the  target  workload  was 
higher.  Once  the  sensor  was  on  target,  it  was  fairly  easy  to  keep  it  on  target. 
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6.  If  at  any  point  you  found  that  your  workload  was  very  different,  what  aspects  of  your  tasks 
(e.g.,  radio  communication,  using  information  to  control  aircraft,  flying  aircraft,  etc.)  were 
affected? 

PI :  (MS  present)  Mission  went  extremely  well. 

P2:  (MS  present)  When  I  had  to  give  direction  to  flying  pilot,  when  we  had  a  target  to  engage. 
P2:  (MS  not  present)  When  the  workload  increased,  I  had  a  hard  time  keeping  the  flying  pilot 
informed  of  tactical  situation. 

P3:  (MS  present)  Using  digital  map  and  sensor  system  to  guide  aircraft  (FP)  and  direct  work  of 
MS. 

P3:  (MS  not  present)  While  in  situation  stated  above  (observing  target  area,  engaging  with 
artillery  and  giving  SITREPs),  operation  of  the  sensor  was  the  most  degraded. 

P4:  (MS  present)  I  don’t  think  anything  was  affected. 

P4:  (MS  not  present)  Radio  communications  became  more  difficult  and  were  lower  on  the 
priority  list.  Communication  between  the  MC  and  FP  became  less  clear  and  hence  positioning 
the  aircraft  was  more  difficult. 

P5:  (MS  present)  All  requirements  became  compressed  into  a  1-2  minute  time  period 
(navigating,  avoiding  enemy,  communicating  with  crew). 

P5:  (MS  not  present)  Monitoring  of  aircraft,  instruments  and  other  non-flying  pilot  duties 
suffered. 

P6:  (MS  present)  Radio  communications  and  battlefield  SA  were  affected  the  most. 

P6:  (MS  not  present)  All  tasks  were  affected.  The  need  to  divide  my  concentration  between  the 
tasks  reduces  the  ability  to  effectively  complete  them. 

P7:  (MS  present)  Most  radio  communications  were  handled  by  the  MS.  There  was  less  time 
available  to  direct  flying  pilot  in  recce  zone. 

P7:  (MS  not  present)  When  doing  radio  communications,  I  felt  I  had  a  better  idea  of  what  was 
being  transmitted.  Guidance  to  FP  suffered  due  to  sensor  operator  duties. 

P8:  (MS  present)  None. 

P8:  (MS  not  present)  Directing  FP.  Situation  awareness:  target  clock  angle  in  relation  to  aircraft. 
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TLX  WORKLOAD  ASSESSMENT 

Instructions.  Place  an  X  on  each  scale  at  the  point  that  represents  the  magnitude  of  each  factor  in 
the  mission  you  just  performed.  Refer  to  the  Workload  Scale  Descriptions  for  definitions  of  each 
factor. 


SEGMENT  OF  MISSION:  Ingress  to  release  point 


MS  present 

MS  not  present 

Mental  demand 

2.7500 

2.6667 

Physical  demand 

2.6250 

2.5833 

Temporal  demand 

2.9167 

2.6667 

Frustration  level 

2.4583 

2.1667 

Effort 

2.6667 

2.7083 

Performance 

8.5000 

8.2500 

SEGMENT  OF  MISSION:  Release  point  to  first  observation  point 


MS  present 

MS  not  present 

Mental  demand 

4.4167 

3.9167 

Physical  demand 

3.0000 

2.8333 

Temporal  demand 

4.2500 

4.3750 

Frustration  level 

3.2083 

3.6250 

Effort 

4.3750 

4.5833 

Performance 

7.8333 

7.6250 
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SEGMENT  OF  MISSION:  Getting  and  maintaining  observation  of  target 


MS  present 

MS  not  present 

Mental  demand 

4.6250 

5.5000 

Physical  demand 

3.0417 

3.7500 

Temporal  demand 

4.7917 

6.0000 

Frustration  level 

4.1667 

5.7083 

Effort 

4.9167 

6.4583 

Performance 

7.7917 

5.9583 

SEGMENT  OF  MISSION:  Overall  activity  while  in  Recce  zone. 


MS  present 

MS  not  present 

Mental  demand 

4.8750 

5.7917 

Physical  demand 

3.4167 

4.0000 

Temporal  demand 

4.5833 

6.3333 

Frustration  level 

4.0833 

6.0000 

Effort 

5.3333 

6.5000 

Performance 

8.2500 

6.3750 
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SEGMENT  OF  MISSION:  Egress 


MS  present 

Ms  not  present 

Mental  demand 

3.0833 

3.4583 

Physical  demand 

2.5417 

3.0000 

Temporal  demand 

3.2083 

3.3333 

Frustration  level 

2.0417 

2.5833 

Effort 

2.7500 

3.2500 

Performance 

8.9583 

8.3750 
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ANNEX  F 

Post-Mission  Comparison  Questionnaires 


120 


TAMSS  SA  Experiment  3 


TAMSS  Experiment  3 

Participant  ID: _ 

Day  1:  Mission  specialist  present  versus  No  Mission  specialist  present 
FLYING  PILOT  VERSION 

Rate  the  impact  of  having  a  Mission  specialist  present  as  compared  to  having  no  Mission 
specialist.  A  rating  of  ‘4’  means  there  was  no  difference  between  a  Mission  specialist  vs.  no 
Mission  specialist. 


COMPARISON 

Mission  specialist  No  Mission 

present  specialist  present 

No 

difference 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

2.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

3.  Awareness  of  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

4.  Awareness  of  activities 

(enemies,  friendly  units) 

1 

2 

3 

4 

5 

6 

7 

5.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

6.  Using  information  from  the 
ERSTA  system  to  control  the 
aircraft 

1 

2 

3 

4 

5 

6 

7 

7.  Awareness  of  heading 

1 

2 

3 

4 

5 

6 

7 

8.  Communication  to  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

9.  Other  radio  communication 

1 

2 

3 

4 

5 

6 

7 

10.  Low-level  flight 

1 

2 

3 

4 

5 

6 

7 

1 1 .  Low-level  maneuvering 

1 

2 

3 

4 

5 

6 

7 

12.  Maintaining  heading 

1 

2 

3 

4 

5 

6 

7 

13.  Overall  workload 

1 

2 

3 

4 

5 

6 

7 
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Participant  ID: _ 

Day  2:  Mission  specialist  present  versus  No  Mission  specialist  present 
FLYING  PILOT  VERSION 

Rate  the  impact  of  having  a  Mission  specialist  present  as  compared  to  having  no  Mission 
specialist.  A  rating  of  ‘4’  means  there  was  no  difference  between  a  Mission  specialist  vs.  no 
Mission  specialist. 


COMPARISON 

Mission  No  Mission 

specialist  specialist  present 

present  No  difference 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

2.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

3.  Awareness  of  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

4.  Awareness  of  activities 

(enemies,  friendly  units) 

1 

2 

3 

4 

5 

6 

7 

5.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

6.  Using  information  from  the 
ERSTA  system  to  control  the 
aircraft 

1 

2 

3 

4 

5 

6 

7 

7.  Awareness  of  heading 

1 

2 

3 

4 

5 

6 

7 

8.  Communication  from  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

9.  Communication  to  Mission 

Commander 

1 

2 

3 

4 

5 

6 

7 

10.  Other  radio  communication 

1 

2 

3 

4 

5 

6 

7 

1 1 .  Low-level  flight 

1 

2 

3 

4 

5 

6 

7 

12.  Low-level  maneuvering 

1 

2 

3 

4 

5 

6 

7 

13.  Maintaining  heading 

1 

2 

3 

4 

5 

6 

7 

14.  Overall  workload 

1 

2 

3 

4 

5 

6 

7 
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Participant  ID:  _ 

Day  1:  Mission  specialist  present  versus  no  Mission  specialist  present 
MISSION  COMMANDER  VERSION 

Rate  the  impact  of  having  a  Mission  Specialist  present  as  compared  to  having  no  Mission 
Specialist  on  YOUR  performance  and  experiences  as  the  MC  in  the  scenarios.  Circle  your 
response  for  each  item.  A  rating  of  ‘4’  means  there  was  no  difference  between  having  a  Mission 
Specialist  vs.  no  Mission  Specialist. 


COMPARISON 

Mission  No  Mission 

specialist  specialist 

present  No  difference  present 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

2.  Keeping  track  of  activity 
(enemies,  friendly  units) 

1 

2 

3 

4 

5 

6 

7 

3.  Overall  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

4.  Using  the  map  to  navigate 

1 

2 

3 

4 

5 

6 

7 

5.  Following  the  route 

1 

2 

3 

4 

5 

6 

7 

6.  Reading  the  digital  map 

1 

2 

3 

4 

5 

6 

7 

7.  Using  the  CDU 

1 

2 

3 

4 

5 

6 

7 

8.  Using  comins 

1 

2 

3 

4 

5 

6 

7 

9.  Communication  to  flying  pilot 

1 

2 

3 

4 

5 

6 

7 

10.  Communication  from  flying 
pilot 

1 

2 

3 

4 

5 

6 

7 

11.  Using  the  sensor  effectively 

1 

2 

3 

4 

5 

6 

7 

12.  Submitting  a  contact  report 

1 

2 

3 

4 

5 

6 

7 

13.  Submitting  a  SITREP 

1 

2 

3 

4 

5 

6 

7 

14.  Executing  a  Fire  Support 

Mission  (FSM) 

1 

2 

3 

4 

5 

6 

7 

15.  Other  radio  communication 

1 

2 

3 

4 

5 

6 

7 

16.  Eyes  out  time 

1 

2 

3 

4 

5 

6 

7 
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17.  Positioning  the  aircraft  in  the 
recce  zone 

18.  Overall  usefulness  of  the 
information  available  from  the 
ERST  A  system 

1 

2 

3 

4 

5 

6 

7 

19.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

20.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

2 1 .  Completing  the  mission 

1 

2 

3 

4 

5 

6 

7 

22.  Overall  workload 

1 

2 

3 

4 

5 

6 

7 
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Participant  ID:  _ 

Day  2:  Mission  specialist  present  versus  no  Mission  specialist  present 
MISSION  COMMANDER  VERSION 

Rate  the  impact  of  having  a  Mission  Specialist  present  as  compared  to  having  no  Mission 
Specialist  on  YOUR  performance  and  experiences  as  the  MC  in  the  scenarios.  Circle  your 
response  for  each  item.  A  rating  of  ‘4’  means  there  was  no  difference  between  having  a  Mission 
Specialist  vs.  no  Mission  Specialist. 


COMPARISON 

Mission  No  Mission 

specialist  specialist 

present  No  difference  present 

1 .  Overall  situation  awareness 

1 

2 

3 

4 

5 

6 

7 

2.  Keeping  track  of  activity 
(enemies,  friendly  units) 

1 

2 

3 

4 

5 

6 

7 

3.  Overall  spatial  orientation 

1 

2 

3 

4 

5 

6 

7 

4.  Using  the  map  to  navigate 

1 

2 

3 

4 

5 

6 

7 

5.  Following  the  route 

1 

2 

3 

4 

5 

6 

7 

6.  Reading  the  digital  map 

1 

2 

3 

4 

5 

6 

7 

7.  Using  the  CDU 

1 

2 

3 

4 

5 

6 

7 

8.  Using  comins 

1 

2 

3 

4 

5 

6 

7 

9.  Communication  to  flying  pilot 

1 

2 

3 

4 

5 

6 

7 

10.  Communication  from  flying 
pilot 

1 

2 

3 

4 

5 

6 

7 

1 1 .  Using  the  sensor  effectively 

1 

2 

3 

4 

5 

6 

7 

12.  Submitting  a  contact  report 

1 

2 

3 

4 

5 

6 

7 

13.  Submitting  a  SITREP 

1 

2 

3 

4 

5 

6 

7 

14.  Executing  a  Fire  Support 

Mission  (FSM) 

1 

2 

3 

4 

5 

6 

7 

15.  Other  radio  communication 

1 

2 

3 

4 

5 

6 

7 

16.  Eyes  out  time 

1 

2 

3 

4 

5 

6 

7 
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17.  Positioning  the  aircraft  in  the 
recce  zone 

18.  Overall  usefulness  of  the 
information  available  from  the 
ERST  A  system 

1 

2 

3 

4 

5 

6 

7 

19.  Overall  mission  awareness 

1 

2 

3 

4 

5 

6 

7 

20.  Anticipating  future  events 

1 

2 

3 

4 

5 

6 

7 

2 1 .  Completing  the  mission 

1 

2 

3 

4 

5 

6 

7 

22.  Overall  workload 

1 

2 

3 

4 

5 

6 

7 
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ANNEX  G 

Summary  Data:  Post-Mission  Comparison  Questionnaires 
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Summary  of  Final  Questionnaire 
TAMMS  Project-  Experiment  3 


Flying  Pilot  Role 


overall  situation  awareness 

3.2857 

overall  mission  awareness 

3.7500 

awareness  of  spatial  orientation 

3.6250 

awareness  of  activities  (enemies,  friendly  units) 

2.8571 

anticipating  future  events 

3.3750 

using  information  from  the  ERSTA  system  to  control  the 
aircraft 

3.1250 

awareness  of  heading 

3.7500 

communication  from  mission  commander 

3.5000 

communication  to  mission  commander 

3.2500 

other  radio  communication 

2.7500 

low-level  flight 

3.7500 

low-level  maneuvering 

3.6250 

maintaining  heading 

4.0000 

1=  Much  better  with  Mission  Specialist,  4=no  difference,  7=  much  better  with  no  Mission 
Specialist 
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Summary  of  Final  Questionnaire 
TAMMS  Project-  Experiment  3 


Mission  Commander  Role 


overall  situation  awareness 

2.0000 

keeping  track  of  activity  (enemies,  friendly 
units) 

1.8750 

overall  spatial  orientation 

3.1250 

using  the  map  to  navigate 

3.3750 

following  the  route 

3.5000 

reading  the  digital  map 

3.6250 

using  the  CDU 

3.0000 

using  coinms 

2.6250 

communication  to  flying  pilot 

2.6250 

communication  from  flying  pilot 

3.0000 

using  the  sensor  effectively 

1.5000 

submitting  a  contract  report 

1.6250 

submitting  a  SITREP 

2.0000 

executing  a  fire  support  mission  (FSM) 

1.6250 

other  radio  communication 

2.2500 

eyes  out  time 

2.2500 

positioning  the  aircraft  in  the  recce  zone 

2.7500 

overall  usefulness  of  the  information 
available  from  the  ERSTA  system 

1.7500 

overall  mission  awareness 

1.8750 

anticipating  future  events 

2.5000 

completing  the  mission 

2.0000 

overall  workload 

2.5000 

1=  Much  better  with  Mission  Specialist,  4=no  difference,  7=  much  better  with  no  Mission 
Specialist 
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List  of  symbols/abbreviations/acronyms/initialisms 


3D 

3  Dimensional 

API 

Application  Programming  Interface 

ASCII 

American  Standard  Code  for  Information  Interchange 

CACR 

Centre  for  Applied  Cognitive  Research 

CDTV 

Colour  Day  Television 

CDU 

Control  Display  Unit 

CMC 

Canadian  Marconi  Company 

COTS 

Commercial  Off-the-Shelf 

CPT 

Cockpit  Procedures  Trainer 

CRM 

Crew  Resource  Management 

CSC 

Communication  Selection  Control 

CSE 

Cognitive  Systems  Engineering 

DACS 

Digital  Audio  Communication  System 

DND 

Department  of  National  Defence 

DRDC 

Defence  Research  &  Development  Canada 

EOS 

Experimenter  Operating  Station 

ERSTA 

Electro-optical  Reconnaissance,  Surveillance  and  Target  Acquisition 

FOM 

Federation  Object  Model 

FOV 

Field-of-View 

FMFS 

Full  Motion  Flight  Simulator 

FP 

Flying  Pilot 

GUI 

Graphical  User  Interface 

HDD 

Head-Down  Display 

HLA 

High  Fevel  Architecture 

HMD 

Helmet  Mounted  Display 

HUD 

Head-Up  Display 

IG 

Image  Generator 

IP 

Internet  Protocol 

ISTS 

Integration  Simulation  Training  System 

LAN 

Focal  Area  Network 

LATEF 

Fand  Aviation  Tactical  Evaluation  Flight 

LCD 

Liquid  Crystal  Display 

M&S 

Modelling  &  Simulation 

MC 

Mission  Commander 

MS 

Mission  Specialist 

MTBF 

Mean  Time  Between  Failures 

NASA 

National  Aeronautics  and  Space  Administration 

NFP 

Non-Flying  Pilot 

NTS 

Networked  Tactical  Simulator 

NVG 

Night  Vision  Goggles 

ODBC 

Open  Database  Connectivity 

OTW 

Out-the-Window 

PC 

Personal  Computer 

PTT 

Part-Task  Trainer 

RMS 

Root  Mean  Square 

RPR 

Real-time  Platform  Reference 

RTOS 

Real  Time  Operating  System 

SA 

Situational  Awareness 

SBE 

Simulation  Based  Evaluation 

SME 

Subject  Matter  Expert 

SQL 

Structured  Query  Language 

TAMSS 

Tactical  Aviation  Mission  Systems  Simulations 

TLX 

Task  Load  Index 

TT 

Tactics  Trainer 

UDP 

Unicast  Datagram  Packet 

WAN 

Wide  Area  Network 
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